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ABSTRACT 


Commercial  and  commercial  off-the-shelf  (COTS)  describe  products  that  have 
been  developed  primarily  for  the  use  of  the  commercial  industries  or  for  the  general 
public.  The  theory  behind  using  commercial  items  in  DoD  applications  is  that  products 
developed  for  commercial  usage  will  be  competitively  priced,  and  because  of  the  large 
user  base,  will  be  very  low  in  unit  price  as  compared  to  a  product  designed  and  developed 
specifically  for  DOD’s  relatively  low -volume  usage.  Also,  in  areas  such  as  computers 
and  software,  commercial  industry’s  speed  of  technology  development  and  acquisition 
surpass  that  which  the  Government  has  been  able  to  achieve. 

As  of  January  2001,  Under  Secretary  of  Defense  (Acquisition,  Technology  & 
Logistics)  (USD(AT&L)),  mandated  an  increase  in  the  utilization  of  commercial  items  as 
follows:  (1)  double  the  dollar  value  of  FAR  Part  12  contract  actions  awarded  in  1999  by 
the  end  of  fiscal  year  2005,  and  (2)  strive  to  increase  the  number  of  FAR  Part  12  contract 
actions  awarded  to  50  percent  of  all  Government  contracts  awarded  by  the  end  of  FY 
2005.  A  great  deal  of  study,  writing  and  analysis  has  gone  into  the  utilization  of 
commercial  item  software  products  but  less  is  known  about  the  utilization  of  commercial 
item  equipment,  which  this  thesis  will  primarily  address. 

It  is  generally  thought  and  accepted  that  the  utilization  of  commercial  equipment 
will  substantially  reduce  cost  and  lead  times  necessary  to  get  systems  fielded.  However, 
there  may  be  some  significant  disadvantages  to  utilization  of  commercial  items  that 
should  be  recognized,  analyzed  and  for  which  corrective  actions  should  be  planned  to 
mitigate  the  implementation  risks. 

The  objective  of  this  thesis  is  to  identify  the  critical  PM  and/or  industry  “issues” 
associated  with  integrating  commercial  items  into  a  DoD  acquisition  program  and  to 
propose  methods  and  process  solutions  that  will  alleviate  many  of  the  implementation 
problems  and  thereby  mitigating  the  implementation  risks. 
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I.  INTRODUCTION 


A,  PURPOSE 

It  is  generally  aeeepted  that  the  utilization  of  commercial  items  will  substantially 
reduce  cost  and  lead  times  necessary  to  get  systems  fielded.  However,  there  may  be 
significant  downsides  and  drawbacks  to  utilization  of  commercial  items  that  should  be 
recognized,  analyzed  and  for  which  corrective  actions  should  be  planned  to  mitigate  the 
implementation  risks. 

The  purpose  and  objective  of  this  thesis  is  to  identify  many  of  the  Program 
Managers’  (PM)  or  industries’  critical  “issues”  associated  with  implementing  and 
integrating  commercial  items  into  a  DoD  program.  The  researcher  will  also  propose 
process  solutions  that  may  alleviate  many  of  the  problems  and  thereby  help  mitigate 
commercial  item  implementation  risks. 

B,  SCOPE  OF  RESEARCH 

The  scope  will  include:  (1)  a  review  of  DoD  regulations  as  they  relate  to 
commercial  items,  (2)  an  in-depth  review  of  the  available  commercial  items  literature,  (3) 
a  compilation  and  evaluation  of  adverse  experiences  by  PMs  and  industries  using 
commercial  items  from  DoD  audit  reports  and  (4)  an  analysis  of  the  audit  results  of  the 
PMs  and  industry  implementing  current  DoD’s  commercial  item  policies.  The  thesis  will 
conclude  with  a  recommendation  of  methods  and  processes  that  work  effectively  when 
utilizing  commercial  items  in  DoD  programs. 

C,  RESEARCH  METHODOLOGY 

The  data  for  this  study  was  obtained  from  several  sources.  First,  the  researcher 
conducted  an  extensive  review  of  available  literature.  This  literature  review  consisted  of  a 
search  of  books,  journal  articles,  microfiche,  the  worldwide  web  and  other  library 
information  resources  pertaining  to  commercial  items.  Finally,  the  researcher  conducted  a 
thorough  review  of  the  Internet  for  current  DoD  policies,  regulations  and  audit  reports 
relating  to  commercial  item  implementation. 

Secondly,  telephone  interviews  were  conducted  with  major  DoD  acquisition 

agencies  involved  in  the  commercial  item  acquisition  process.  These  agencies  included 
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Under  Secretary  of  Defense  for  Acquisition  Logistics  and  Technology,  USD(AL&T), 
Deputy  Under  Secretary  of  Defense  for  Acquisition  Reform,  DUSD(AR),  Assistant 
Secretary  of  the  Army  for  Acquisition  Technology  and  Logistics,  ASA(AT&L),  and 
Headquarters  Army  Material  Command,  HQAMC,  where  it  was  determined  that  there 
was  no  current,  consolidated,  PM  commercial  item  usage  data  available.  Telephone 
interviews  were  also  conducted  with  a  small  number  of  PM  and  industry  personnel  in 
order  to  identify  current  problems  and  issues  being  experienced  with  commercial  item 
utilization. 

Thirdly,  a  consolidated  data  call  was  obtained  from  Communication  Electronics 
Command  (CECOM),  which  contained  issue  papers  from  three  Program  Managers  (PM); 
PM-Physical  Security  Equipment,  (PM-PSE);  PM-Defense  Communications  and  Army 
Switched  Systems,  (PM-DCASS);  and  PM-Global  Positioning  System,  (PM-GPS);  and 
from  the  Competition  Management  Division  Eegal  Office.  [SMC] 

Einally,  additional  data  was  obtained  from  General  Accounting  Office  (GAO), 
Army  Audit  Agency  (AAA),  and  Department  of  Defense  Inspector  General  (DoD  IG) 
audit  reports. 

D,  RESEARCH  QUESTIONS 

To  achieve  the  objectives  of  this  study,  the  primary  research  question  was  as  follows: 

What  are  the  most  important  problems  and  solutions  associated  with  the  use  of 
commercial  products  within  defense  acquisition? 

Prom  the  basic  research  question,  the  following  subsidiary  questions  were  developed; 

1 .  What  is  the  definition  of  commercial  items  and  their  relevance  to  DoD? 

2.  What  are  DoD’s  requirements  and  objectives  regarding  commercial  item 
usage? 

3.  What  are  the  critical  problems  or  issues  confronted  by  PMs  utilizing 
commercial  items? 

4.  How  can  PM  problems  or  issues  utilizing  commercial  items  be  resolved? 
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E,  ORGANIZATION  OF  STUDY 

The  researcher  will  first  review  the  background  of  commercial  items  and  the  DoD 
commercial  item  requirements  in  some  detail  in  Chapter  II.  In  Chapter  III,  the  Program 
Managers’  and  industries’  experiences  utilizing  commercial  products  within  their  DoD 
programs  will  be  identified  and  categorized  for  further  study.  In  Chapter  IV,  the  PM  and 
industry  critical  problems  and  issues  are  analyzed.  In  Chapter  V,  potential  process 
solutions  will  identify  the  best  practices  and  conclude  with  areas  for  further  research. 
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II.  BACKGROUND 


A,  COMMERCIAL  ITEMS  DEFINED 

The  Federal  Aequisition  Regulation  (FAR)  defines  the  terms  “eommercial  item” 
and  “nondevelopmental  item”  as  follows: 

1 .  Commercial  Item 

a.  Any  item,  other  then  real  property,  that  is  of  a  type 
customarily  used  for  nongovernmental  purposes  and  that 
has  been  sold,  leased,  or  licensed  to  the  general  public,  or 
has  been  offered  for  sale,  lease  or  license  to  the  general 
public. 

b.  Any  item  that  evolved  from  an  item  described  in  paragraph 
a,  above  through  advances  in  technology  or  performance 
that  is  not  yet  available  in  the  commercial  market  in  time  to 
meet  the  delivery  requirements  of  the  solicitation;  but  will 
be  available  in  the  commercial  marketplace  in  time  to 
satisfy  the  delivery  requirements  under  a  Government 
solicitation. 

c.  Any  item  that,  but  for  modifications  of  a  type  customarily 
available  in  the  commercial  market  or  minor  modifications 
made  to  meet  DoD  requirements,  would  satisfy  the  criteria 
in  paragraph  a  or  b,  above,  etc. 

[FAR  Part  2] 

2.  Nondevelopmental  Item 

Any  previously  developed  item  used  exclusively  for 
governmental  purposes  by  a  Federal  agency,  a  State  or 
local  government,  or  a  foreign  government  with  which  the 
U.S.  has  a  mutual  defense  cooperation  agreement. 

[FAR  Part  2] 


Because  the  specific  emphasis  within  DoD  is  to  increase  the  utilization  of 
commercial  items,  and  nondevelopmental  items  (NDI)  are  placed  second  in  the  priority 
of  implementation,  this  thesis  will  only  address  commercial  item  usage.  NDI  literature 
will  be  reviewed  inasmuch  as  problems  experienced  utilizing  NDI,  will  for  the  most  part, 
apply  to  commercial  item  usage  as  well.  Also,  there  is  a  great  deal  of  indiscriminate  and 
imprecise  terminology  usage.  Thus,  when  the  term  “NDI”  is  used,  the  literature  may 
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actually  be  referring  to  a  eommereial  item.  The  speeifie  study  of  nondevelopmental  items 
will  be  exeluded  from  this  study,  but  many  of  the  problems  and  issues  found  when 
utilizing  eommereial  items  also  apply  when  using  nondevelopmental  items.  Thus,  the 
same  types  of  preeautionary  and  risk-redueing  taeties  as  reeommended  in  this  thesis  for 
eommereial  items  may  be  used  when  aequiring  nondevelopmental  items. 

Government  Off-The-Shelf  (GOTS)  is  also  a  eommonly  used  term  for 
nondevelopmental  items  (NDI)  that  are  government-unique  items  in  use  by  federal,  state 
or  other  governmental  ageney  or  by  a  foreign  government  with  whieh  the  United  States 
has  a  mutual  defense  eooperation  agreement.  This  area  will  also  be  exeluded  and  this 
term  will  not  be  utilized  in  this  thesis. 

Commereial  Off-The-Shelf  (COTS)  is  a  familiar  and  well-used  term  within 
industry  and  DoD.  However,  there  is  no  definition  of  COTS  in  the  FAR  Part  2.  As  used, 
COTS  is  traditionally  defined  as  its  name  implies,  i.e.,  a  eommereial  item  that  is 
purehased  off  the  eommereial  shelf  without  any  modifieation  whatsoever.  This  term  will 
not  be  used  in  this  thesis  exeept  where  quoted,  in  favor  of  the  broader  term  Commereial 
Item  that  ineludes  COTS  as  a  subset. 

B,  BROADENED  DEFINITION  OF  COMMERCIAL  ITEMS 

The  Federal  Aequisition  Streamlining  Aet  (FASA)  of  1994  and  the  Federal 
Aequisition  Reform  Aet  (FARA)  of  1996  broadened  the  definition  of  a  eommereial  item. 
As  they  redefined  it,  systems,  subsystems,  and  eomponents  other  than  real  property  are 
eonsidered  eommereial  items  or  modified  eommereial  items  if  they  are  of  a  type  that  ean 
be  eategorized  into  one  of  the  following  areas: 

1 .  Has  been  offered  for  sale,  sold,  leased,  or  lieensed  to  the  general  publie. 

2.  Evolved  through  teehnology  or  performanee  advanees  and  will  be 
available  in  the  eommereial  marketplaee  in  time  to  meet  proposed  Government  delivery 
requirements. 

3.  Required  modifieation  of  a  type  eustomarily  available  in  the  eommereial 
marketplaee  or  required  minor  modifieation  of  a  type  not  customarily  available  in  the 
eommereial  marketplaee  to  meet  DoD  requirements. 
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Besides  broadening  the  definition  of  eommereial  items,  the  1994  FAS  A  and  1996 
FARA  Acts  enabled  DoD  to  revise  and  modify  acquisition  regulations.  DoD  was  in  a 
position  to: 

1 .  Simplify  contracting  procedures  for  acquisitions  not  costing  more  than  $5 
million,  when  contracting  officers  reasonably  believe  that  only  commercial  items  will  be 
offered; 

2.  Exempt  the  contract  requirement  to  provide  cost  or  pricing  data  for 
commercial  items; 

3.  Give  contracting  officers  flexibility  in  determining  price  reasonableness 
for  contracts,  subcontracts,  and  modifications; 

4.  Provide  contracting  officers  a  list  of  Federal  Acquisition  Regulation 
(FAR)  provisions  that  are  not  applicable  for  commercially  available  items;  and 

5.  Remove  more  than  30  contractor  certification  requirements  that  are  not 
specifically  imposed  by  statue. 

[FARA,  FASA] 

C.  COMMERCIAL  ITEMS  CLARIFIED 

On  January  5,  2001,  Dr.  J.  S.  Gansler,  then  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics,  issued  a  policy  memorandum  to  clarify  and  to 
help  overcome  some  of  the  barriers  being  experienced  within  the  Department  of  Defense 
in  utilizing  commercial  items.  An  Integrated  Process  Team  (IPT)  had  been  formed  at  his 
direction  and  was  headed  by  both  the  Deputy  Under  Secretary  of  Defense  for  Acquisition 
Reform  (DUSD  (AR))  and  the  Director  of  Defense  Procurement.  The  IPT  was  chartered 
to  review  DoD  commercial  item  determinations  and  evaluate  whether  additional 
guidance,  tools,  or  training  were  necessary.  Dr.  Gansler’ s  memorandum  says  that  the  IPT 
found  “inconsistent  commercial  item  determination  and  weak  market  research  among  the 
obstacles  that  exist  to  broadening  the  use  of  commercial  items  within  the  DoD.”  [DoD. 
USD(AT&L),  5  January  2001] 

Dr.  Gansler’s  memorandum  also  provided  clarifying  definitions  of  FAR  Part  12 
for  greater  consistency  within  DoD.  Three  of  the  most  important  of  these  are  as  follows: 

Commercial  Off-the-Shelf  (COTS):  A  product  does  not  have  to  be 

commercial-off-the-shelf  (COTS)  to  meet  the  “commercial  item” 
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definition.  COTS  items  are  a  subset  of  commereial  items.  The  eommereial 
item  definition  is  mueh  broader  than  produets  that  are  presently  available 
off-the-shelf.  It  includes  items  that  have  only  been  “offered”  for  sale, 
lease,  or  license  to  the  general  public,  as  well  as  those  that  have  evolved 
from  a  commercial  item  and  are  offered  for  sale,  even  if  not  yet  available 
in  the  commercial  marketplace.  However,  evolved  items  must  be  available 
in  the  commercial  marketplace  in  time  to  satisfy  solicitation  delivery 
requirements.  In  addition,  all  other  elements  of  the  commercial  item 
definition  at  FAR  2.101  must  also  be  met. 

Modified  Commercial  Items:  When  items  available  in  the  commercial 
market  cannot  meet  the  Department’s  need,  DoD  must  determine  whether 
market  items  can  be  or  have  been  modified  so  that  FAR  Part  12  can  be 
used.  Two  types  of  modifications  are  available:  (1)  modifications  of  a 
type  available  in  the  commercial  marketplace;  and,  (2)  minor 
modifications  of  a  type  n^  customarily  available  in  the  commercial 
marketplace  made  to  Federal  Government  requirements.  For  modifications 
of  a  type  available  in  the  commercial  marketplace,  the  size  or  extent  of 
modifications  is  unimportant.  For  minor  modifications,  the  item  must 
retain  a  predominance  of  nongovernmental  functions  or  physical 
characteristics. 

“Of  a  Type”:  The  phrase  “of  a  type”  is  not  intended  to  allow  the  use  of 
FAR  Part  12  to  acquire  sole-source,  military  unique  items  that  are  not 
closely  related  to  items  already  in  the  marketplace.  Instead,  “of  a  type” 
broadens  the  commercial  item  definition  so  that  qualifying  items  do  not 
have  to  be  identical  to  those  in  the  commercial  marketplace.  The  best 
value  offer  in  a  competitive  Part  12  solicitation  can  be  an  item  that  has 
previously  satisfied  the  Government’s  need  but  has  not  been  sold,  leased, 
licensed,  nor  offered  for  sale,  lease  or  license  to  the  general  public  (a 
nondevelopmental  item  as  defined  in  10  USC  403  (13).  In  this  scenario, 
the  phrase  “of  a  type”  allows  the  best  value  offer  to  qualify  for  a  Part  12 
contract  as  long  as  it  is  sufficiently  like  similar  items  that  meet  the 
government’s  requirement  and  are  sold,  leased,  licensed,  or  offered  for 
sale,  lease  or  license  to  the  general  public.  In  such  instances,  “of  a  type” 
broadens  the  statutory  commercial  item  definition  to  allow  Part  12 
acquisition  of  a  government-unique  item  that  can  compete  with 
commercial  items  that  meet  the  government’s  requirement.  This  avoids  the 
undesirable  result  of  shutting  out  otherwise  price-competitive  preexisting 
suppliers  of  government-unique  items  from  Part  12  solicitations. 

[DoD.  USD(AT&L),  5  January  2001] 

Since  COTS  has  been  defined  as  a  subset  of  “commercial  items”  and  Dr. 

Gansler’s  memorandum  specifically  addresses  the  broader  scope  of  commercial  items. 


8 


this  researcher  will  use  the  term  “commercial  item(s)”  throughout  this  thesis  and  the  term 
commercial  off-the-shelf  (COTS)  will  not  be  used. 

A  memorandum  titled  Commercial  Acquisitions,  dated  26  March  2001,  was 
issued  by  the  Acting  Assistant  Secretary  of  the  Army  for  Acquisition,  Logistics  & 
Technology,  incorporating  a  plan  titled,  ^"Implementation  Plan  for  Increasing  the  Use  of 
FAR  Part  I2'\  The  plan  announced  a  policy  that  all  contractually  provided  services  (with 
the  exception  of  services  under  FAR  Part  36)  were  presumed  to  be  commercial  and  that 
FAR  Part  12  policies  and  procedures  would  be  used  to  buy  those  services.  The  plan 
further  states,  “For  those  services  where  the  results  of  market  research  indicate  that  the 
service  is  not  commercial,  the  local  Competition  Advocate  must  approve  the  commercial 
determination.”  (ASAAL&T)  This  plan  puts  even  more  emphasis  on  the  utilization  of 
commercial  items  within  the  Army,  which  may  open  the  way  for  all  DoD. 

D,  DOD  REGULATION  5000.1:  COMMERCIAL  ITEM  REQUIREMENT 

Government  regulation  echoes  the  statutory  preference  for  commercial  items  and 
indicates  their  importance  to  DoD.  The  best  evidence  of  this  is  DoDR  5000.1,  as  noted 
below. 


In  response  to  user  requirements,  priority  consideration  shall  always  be 
given  to  the  most  cost-effective  solution  over  the  system’s  life  cycle.  In 
general,  decision-makers,  users,  and  program  managers  shall  first  consider 
the  procurement  of  commercially  available  products,  services,  and 
technologies,  or  the  development  of  dual-use  technologies,  to  satisfy  user 
requirements,  and  shall  work  together  to  modify  requirements,  whenever 
feasible,  to  facilitate  such  procurements.  Market  research  and  analysis 
shall  be  conducted  to  determine  the  availability,  suitability,  operational 
supportability,  interoperability,  and  ease  of  integration  of  existing 
commercial  technologies  and  products  and  of  non-developmental  items 
prior  to  the  commencement  of  a  development  effort. 

[DoD  Directive  5000.1,  paragraph  4.2.3] 

E,  MARKET  RESEARCH 

Market  research  is  the  process  of  collecting  and  analyzing  data  about  the  products 
and  technologies  available  in  the  marketplace.  The  market  research  IPT  then  determines 
which  product  or  technology  satisfies  the  agencies  procurement  needs.  Market  research 
has  been  a  statutory  requirement  for  almost  20  years,  since  the  passage  of  the 
Competition  in  Contracting  Act  of  1984.  This  act  requires  the  use  of  market  research  and 
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procurement  planning  to  promote  the  use  of  eompetitive  proeedures  in  federal 
eontraeting. 

In  November  1990,  Congress  reemphasized  market  researeh  for  the  DoD  in  the 
National  Defense  Authorization  Aet  for  Fiseal  Year  1991  (P.L.  101-510).  The  goal  of 
this  aet  was  to  eneourage  DoD  to  save  money  and  reduee  aequisition  eyele  time  by 
buying  produets  that  were  eommereially  available  or  had  already  been  developed. 

The  Federal  Aequisition  Streamlining  Aet  (FASA)  of  1994  stipulated  that  all 
federal  exeeutive  ageneies  must  eonduet  market  researeh  before  developing  new 
speeifieations  for  proeurement  or  solieiting  bids  for  eontraets  exeeeding  $100,000. 
FASA  (Seetion  8104,  paragraph  2377  of  Publie  Law  103-355)  requires  heads  of  ageneies 
to  use  the  results  of  market  researeh  to  determine  whether  there  are  eommereial  items 
available  that  meet  the  ageney’s  requirements,  if  those  requirements  were  modified  to  a 
reasonable  extent.  For  example,  if  a  requirement  speeifies  that  an  aireraft  must  meet  a 
maximum  speed  of  100  mph  but  there  is  a  eommereial  item  available  with  maximum 
speed  of  90  mph,  the  requirements  eommunity  must  seriously  eonsider  redueing  that 
requirement  so  that  the  eommereial  item  ean  be  used.  If  suitable  eommereial  items  are 
not  available,  the  ageney  will  then  use  nondevelopmental  items. 

The  FAR  implemented  those  market  researeh  provisions  and  essentially 
established  market  researeh  as  a  tool  for  identifying  sourees  to  ensure  eompetition  as  well 
as  eommereial  produets  to  meet  an  ageney’s  needs.  FAR  Part  10  requires  that  ageneies 
eonduet  market  researeh  appropriate  to  the  eireumstanees  before  developing  new 
aequisition  requirements  doeuments.  If  market  researeh  establishes  that  a  eommereial 
item  eannot  fill  the  Government’s  need,  ageneies  are  required  by  FAR  10.002(e)  to 
reevaluate  the  requirement  for  possible  restatement  to  enable  use  of  eommereial  or 
nondevelopmental  items,  as  defined  in  FAR  2.101.  The  findings  of  market  researeh  must 
be  doeumented  per  FAR  10.002  (e). 

F.  COMMERCIAL  ITEM  USAGE  CONCEPT 

The  eoneept  behind  greatly  expanding  the  usage  of  eommereial  items  in  DoD 
applieations  is  best  explained  in  a  study  produeed  by  the  USD(AT&L),  as  follows: 
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Expanding  the  use  of  eommereial  items  in  Department  of  Defense  (DoD) 
systems  offers  the  DoD  opportunities  for  redueed  eyele  time,  faster 
insertion  of  new  teehnology,  lower  life  eyele  eosts,  greater  reliability  and 
availability,  and  support  from  a  more  robust  industrial  base. 

[DoD.  ODUSD(AR),  14  July  2000] 


Beeause  of  the  eompetitive  pressures  experieneed  by  eommereial  manufaetures  to 
remain  in  business,  they  are  foreed  to  develop  and  produee  the  best  possible  quality 
produets  at  the  most  eompetitive  eost.  DoD,  therefore,  will  reap  the  benefits  and  get  the 
best  value  without  expending  the  time  and  funds  neeessary  to  develop,  test  and  field  a 
similar  produet.  Also,  in  areas  sueh  as  eomputers  and  software,  the  speed  of  teehnologieal 
development  and  aequisition  by  eommereial  industry  surpasses  that  whieh  the 
Government  has  been  able  to  aehieve.  Therefore,  eommereial  items  will  save  the  DoD 
both  time  and  money,  as  well  as  keeping  them  teehnologieally  eompetitive. 

Clearly,  the  mandate  of  the  USD(AT&L)  is  for  DoD  to  dramatieally  inerease  the 
utilization  of  eommereial  items  to  improve  the  aequisition  proeess  by  speeding  it  up, 
redueing  eosts,  and  keeping  paee  with  eommereial  teehnology  development.  As  the 
USD(AT&L)  memorandum  says: 


To  the  maximum  extent  possible,  eommereial  aequisitions  should  be 
eondueted  using  FAR  Part  12.  The  use  of  FAR  Part  12  is  designed  to 
provide  the  Department  of  Defense  (DoD)  with  greater  aeeess  to 
eommereial  markets  with  inereased  eompetition,  better  priees,  and  new 
market  entrants  and/or  teehnologies. 

[DoD.  USD(AT&F),  5  January  2001] 

G.  DOD  COMMERCIAL  ITEM  REQUIREMENTS 

USD(AT&F’s)  memorandum  of  January  5,  2001  provided  further  speeifie 
eommereial  item  aequisition  dollar-value  goals  for  DoD  to  meet.  The  two  most  important 
of  these  are  quoted  as  follows: 

1 .  Eaeh  Serviee  and  Defense  Ageney  should  double  the  dollar  value 
of  FAR  Part  12  eontraet  aetions  awarded  in  1999  by  the  end  of  fiseal  year 
(FY)  2005.  This  would  bring  the  DoD  total  FAR  Part  12  eontraet  actions 
from  $12.6  billion  to  $25.2  billion.* 
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2.  Each  Service  and  Defense  Ageney  should  strive  to  inerease  the 
number  of  FAR  Part  12  eontraet  aetions  awarded  to  50  pereent  of  all 
Government  eontraet  awarded  by  the  end  of  FY  2005.* 

*For  purposes  of  these  goals,  a  eontraet  aetion  is  defined  as  any  new 
eontraet  award  and/or  new  delivery  order  placed  against  a  eontraet 
awarded  with  a  value  greater  than  $25,000. 

[DoD.  USD(AT&F),  5  January  2001] 

In  support  of  these  objectives.  Dr.  Gansler’s  memorandum  tasked  the  DFISD(AR) 
to  traek  the  Department  of  Defense  progress  in  meeting  these  goals.  He  also  requested 
that  the  DUSD(AR)  and  the  Direetor  of  Defense  Proeurement  eharter  an  Integrated 
Proeess  Team  (IPX)  to:  “determine  the  feasibility  of  establishing  a  pilot  program  so  that 
the  Serviees  and  Ageneies  may  eolleet  market  researeh  and  Commereial  Item 
Determinations  in  a  eentral  database,  or  developing  tools  to  assist  in  ensuring  eommereial 
item  determinations  are  reasonably  consistent.”  [DoD.  USD(ATF),  5  January  2001] 

It  is  elear  that  it  is  DoD  poliey  to  strongly  eneourage  the  utilization  of  eommereial 
items  in  the  DoD  aequisition  proeess.  In  support  of  using  eommereial  items  and  in  order 
to  shorten  the  time  it  takes  to  eomplete  an  aequisition,  the  proeess  of  trading  user  “gold 
plated”  requirements  for  eommereial  items  is  an  option  that  must  be  strongly  eonsidered. 
However,  any  sueh  tradeoffs  must  be  eritieally  examined  in  order  that  real  user  needs, 
neeessary  to  meet  emerging  threats,  are  not  saerifieed  for  the  eommereial  item  goal. 

H.  USING  COMMERCIAL  ITEMS 

The  use  of  eommereial  items  in  DoD  is  not  new;  many  programs  have  used 
eommereial  items  to  some  extent.  For  example,  a  vehiele  program  might  use  eommereial 
tires  and  batteries,  with  everything  else  developed  espeeially  for  the  Government 
program.  Or  a  eomplete  program,  sueh  as  a  truek,  might  be  a  slightly  modified 
eommereial  item.  The  extent  of  eommereial  item  usage  varies  from  program  to  program. 
With  the  new  emphasis  on  using  eommereial  items  within  DoD,  a  new  paradigm  of 
system  aequisition  thinking  is  being  ereated.  The  2001  system  aequisition  management 
proeess  graphie  representation  from  DoDI  5000.2  is  shown  in  Figure  1. 
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Figure  1.  Defense  Acquisition  Management  Framework 

(From:  DoDI  5000.2) 


The  model  proeess  is  separated  into  three  segments:  pre-systems  aequisition, 
systems  acquisition,  and  sustainment.  This  process  is  then  broken  down  contextually,  by 
OSD,  into  the  Traditional  Model,  as  shown  in  Figure  2.  The  Traditional  Model  consists 
of  three  segments  as  follows:  1)  System  Context,  which  includes  the  requirements 
definition  and  concept  exploration  phases  of  systems  acquisition,  2)  Architecture  & 
Design,  which  includes  the  design  &  development  phase  of  systems  acquisition,  and  3) 
Implementation,  which  includes  the  building,  fielding  and  supporting  phases  of  the 
acquisition  system.  This  Traditional  Model  is  then  transposed  into  the  Recommended 
Model,  which  overlaps  the  three  areas  above  and  overlays  the  process  of  Simultaneous 
Definition  and  Tradeoffs  as  the  new  paradigm  for  increased  utilization  of  commercial 
items  within  DoD  programs.  As  the  OSD  says,  “increased  reliance  on  commercial  items 
implies  a  different  paradigm  of  systems  acquisition.”  [DoD.  ODUSD(AR),  pg  3,  14  July 
2000] 

The  greatest  difference  between  the  two  models  shown  in  Figure  2  is  that  greater 

interaction  is  required  between  the  system  context,  architecture  and  design  segments 
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when  using  commercial  items,  as  shown  by  the  overlapping  of  contextual  areas.  This 
new  model  indicates  that  there  is  a  greater  need  for  use  of  Integrated  Product/Process 
Development  (IPPD)  and  IPTs  as  the  overlapping  areas  indicate.  As  the  OSD  states: 
“Managing  this  interaction  requires  unprecedented  cooperation  among  the  program 
office,  the  stakeholders,  the  contractor,  and  in  many  cases  the  vendor  in  order  to  effect 
the  tradeoffs  necessary  to  keep  the  program  on  track.”  [DoD.  ODUSD(AR),  pg  3,  14  July 
2000] 

Traditional  Model  Recommended  Model 


Figure  2.  Traditional  Model  vs.  Recommended  Model 
(From:  ODUSD(AR),  pg  3,  7/14/2000) 


PMs  must  not  only  manage  a  traditional  system  but  now  must  also  integrate  the 
selection  of  commercial  items  in  varying  degrees  into  their  program,  along  with  all  of  the 
problems  associated  with  commercial  item  usage.  Many  acquisition  programs  have  had 
problems  or  have  failed  due  to  a  lack  of  consideration  for  the  uniqueness  of  commercial 
item  usage  in  their  programs,  as  well  as  inattention  to  the  differences  noted  below. 

Some  of  the  most  important  new  paradigm  considerations  and  issues  a  PM  must 
deal  with  are  as  follows: 
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1,  The  marketplace,  not  the  PM,  drives  development  of  the  commercial 
product. 

The  commercial  marketplace  is  driven  by  customer  demand,  customer  satisfaction 
and  competition.  Because  DoD’s  demand  for  commercial  products  is  such  a  small  portion 
of  total  commercial  demand,  DoD  cannot  dictate  development  to  commercial  producers. 
Using  software  as  an  example,  it  is  noted  that,  “DoD  accounts  for  less  than  1%  of  the 
U.S.  market,  and  less  than  0.6%  of  the  international  market.”  [Anderson,  pg.  4,  1998] 

2,  A  change  of  the  commercial  product  (which  is  almost  a  certainty)  will 
drastically  affect  program  supportahility. 

Change  is  the  norm  in  the  commercial  item  marketplace,  and  must  be  carefully 
planned  for  by  PMs  when  using  commercial  items  in  their  programs.  Planning  for 
program  supportahility  over  the  expected  life  cycle  of  the  product  is  critical.  A  method 
for  handling  obsolescence  should  be  devised,  such  as  preplanning  product  improvement 
or  upgrading  commercial  items  with  new  technology  insertion  along  with  retirement  of 
the  obsolete  items.  Another  method  cited  by  ODUSD(AR)  was  “by  adopting  an  open 
system  architecture  with  modular  designs,  maintaining  close  relationships  with 
commercial  item  vendors,  and  monitoring  the  marketplace,  one  particularly  successful 
program  not  only  avoided  technological  obsolescence,  but  also  developed  a  “sparing” 
model  that  reduced  the  cost  of  spare  components  by  40%. ”  [DoD.  ODUSD(AR),  pg.  17, 
14  July  2000] 

3,  Integrating  a  commercial  item  into  a  program  is  fundamentally 
different  from  developing  a  custom  product,  and  may  require  similar  PM  attention. 

In  the  development  of  a  custom  product,  the  PM  dictates  the  behavior  of 
components  and  interfaces  among  components.  However,  as  ODUSD(AR)  indicates, 
“Program  managers  who  use  commercial  items  have  little  insight  into  how  the 
commercial  items  are  put  together,  how  they  behave,  and  why.”  [DoD.  ODUSD(AR),  pg. 
5,14  July  2000] 

4,  Greater  teamwork  among  stakeholders  during  the  selection  of 
commercial  items  from  the  marketplace  is  essential  to  keep  the  program  on  track. 

There  is  a  dynamic  interaction  necessary  between  stakeholders  when  using 
commercial  items  in  order  to  adequately  perform  commercial  item  market  research, 
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determination  and,  if  neeessary,  modifieation  of  user  requirements,  establishment  of 
eommereial  item  logistic  support  requirements  and  determination  of  system  reliability. 
As  ODUSD(AR)  states,  “Managing  this  interaction  requires  unprecedented  cooperation 
among  the  program  office,  the  stakeholders,  the  contractor,  and  in  many  cases  the  vendor 
in  order  to  effect  the  tradeoffs  necessary  to  keep  the  program  on  track.”  [DoD. 
ODUSD(AR),  pg.  3,  14  July  2000] 

5.  There  must  be  a  willingness  to  adapt  the  “requirements”  to  the 
capabilities  available  in  the  marketplace  rather  than  adapting  the  commercial 
capabilities  to  the  DoD  “requirements.” 

Paring  down  requirements  to  those  absolutely  essential  to  the  program,  as 
opposed  to  those  “nice  to  have,”  will  allow  for  requirements  flexibility  to  enable 
choosing  an  acceptable  commercial  item. 

ODUSD(AR)  notes  that,  “Requirements  specification  must  be  flexible  and 
negotiable.  A  traditional  development  model  that  specifies  all  system  requirements  prior 
to  considering  the  capabilities  available  in  the  marketplace  is  ill  suited  to  the 
development  of  systems  incorporating  commercial  items.”  [DoD.  ODUSD(AR),  pg.  8,  14 
July  2000] 

6.  Commercial  item  usage  means  that  many  of  the  normal  acquisition 
activities  (such  as  system  development  and  sustainment)  will  be  repeated  throughout 
the  lifecycle  of  the  program,  because  a  commercial  item  is  never  really  complete 
(i.e.,  it  is  always  changing). 

Frequent  change  in  the  commercial  item’s  configuration  will  mean  constant 
monitoring  of  the  impact  to  the  program,  as  indicated  by  ODUSD(AR):  “However,  new, 
changed,  and  obsolete  commercial  items  necessitate  repeated  cycles  of  requirement 
definition,  commercial  item  evaluation,  and  system  engineering.  Some  form  of 
replanning  and  reengineering  will  be  ongoing  throughout  the  life  of  the  system.”  [DoD. 
ODUSD(AR),  pg.  5,  14  July  2000] 

7.  The  product  must  be  tested  or  verified  to  ensure  that  required 
capabilities  are  in  fact  met  as  promised  by  the  commercial  item  manufacturer, 

A  commercial  item’s  capability  may  not  always  be  as  stated,  and  should  always 
be  verified.  As  noted  by  ODUSD(AR),  “Another  program  that  was  using  multiple 
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commercial  items  found  that  even  basie,  advertised  eapabilities  of  eommereial  items  had 
to  be  tested  before  the  program  eould  begin  its  planned  integration  testing.”  [DoD. 
ODUSD(AR),  pg.  18,  14  July  2000] 

I.  POTENTIAL  BENEFITS  OF  USING  COMMERCIAL  ITEMS 

There  are  unique  obstaeles  when  using  eommereial  items.  However,  when 
addressed  correetly,  the  benefits  can  far  outweigh  the  risks.  Programs  have  doeumented 
great  savings  of  time  and  money  when  the  unique  challenges  encountered  by  the  use  of 
eommereial  items  have  been  adequately  addressed.  However,  the  use  of  eommereial 
items  does  not  reduee  or  eliminate  the  risks  assoeiated  with  the  traditional  aequisition 
proeesses.  It  is  eritieal  that  the  overall  system  be  guided  through  all  the  traditional 
aequisition  proeesses  of  engineering,  development,  integration,  testing,  delivery  and 
sustainment,  and  that  eaeh  program  be  well  managed.  As  the  ODUSD(AR)  states,  “While 
there  are  signifieant  benefits,  these  benefits  can  be  attained  only  by  understanding  and 
addressing  the  signifieant  new  ehallenges  that  are  driven  by  the  fundamental  differenees 
between  building  items  and  buying  them.”  [DoD.  ODUSD(AR),  pg.  20,  14  July  2000] 
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III.  PM  &  INDUSTRY  COMMERCIAL  ITEM  USAGE 
EXPERIENCE  &  PROBLEMS 

A.  INTRODUCTION 

This  chapter  will  document  many  of  the  PM  and  industry  “issues”  assoeiated  with 
implementing  and  integrating  eommereial  items  into  a  DoD  program.  These  issues  eome 
from  several  sources,  sueh  as  the  Army  Audit  Ageney,  General  Aceounting  Office 
(GAO),  DoD  Inspeetor  General  (DoD  IG),  and  data  from  a  CECOM  PM. 

Issues  will  be  presented  by  audit  ageney  report  or  issue  paper  topic  and  then 
segregated,  when  applieable,  into  the  aequisition  topic  areas  as  follows:  needs 
determination;  aequisition  strategy;  market  investigation;  eoneept  formulation;  risk 
analysis;  management  of  integration;  test  and  evaluation;  logisties  support,  ineluding 
reliability;  support  requirements;  and  teehnieal  data  for  eompetitive  reproeurement. 

B,  BACKGROUND  INFORMATION 

In  many  of  the  refereneed  Army  doeuments  that  follow,  the  term 
“nondevelopmental  item”  is  used  synonymously  with  the  term  “eommereial  item”  as 
previously  defined  in  this  thesis.  Where  the  term,  “nondevelopmental  item”  is  used  in  the 
doeumentation,  it  will  be  replaeed  with  “eommereial  item”  throughout  this  portion  of  the 
research,  except  when  “nondevelopmental  item”  is  used  in  a  quotation. 

Sinee  understanding  the  issues  presented  herein  requires  some  understanding  of 
the  aequisition  proeess  areas  discussed,  basie  definitions  are  provided  below  for  eaeh  of 
the  following  eoneepts:  requirements  determination;  aequisition  strategy;  market 
investigation;  eoneept  formulation;  management  of  integration;  test  and  evaluation;  and 
logisties  support,  whieh  includes:  reliability,  support  requirements,  and  teehnieal  data  for 
eompetitive  reproeurement. 

1.  Acquisition  Strategy 

All  major  material  aequisition  programs  require  an  aequisition  strategy.  In  the 
Army,  for  example,  the  material  developer  prepares  the  aequisition  strategy  during  the 
eoneept  exploration  phase,  prior  to  being  approved  at  the  milestone  I/A  decision  point. 
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The  acquisition  strategy  documents  the  approach  for  satisfying  a  requirement  that 
includes  plans  and  justifications  for  streamlining  the  acquisition  process  -  that  is,  plans  to 
shorten  or  eliminate  any  of  the  acquisition  phases.  The  goal  of  the  material  developer  is 
to  tailor  the  strategy  to  create  the  shortest  path  through  the  acquisition  process  that  will 
successfully  satisfy  the  material  requirement. 

If  the  acquisition  strategy  is  not  accurately  identified  early  in  the  process, 
incorrect  phases  of  the  acquisition  process  will  be  applied  to  one  of  the  following:  the 
system,  the  part,  the  component,  or  to  the  integration  of  all  these  into  a  system.  Any 
streamlining  that  occurs  which  eliminates  acquisition  phases  or  steps  in  the  commercial 
item  acquisition  process  must  be  fully  addressed  in  the  acquisition  strategy.  The 
treatment  of  basic  low-risk,  off-the-shelf  commercial  items  as  medium  or  high-risk 
commercial  items  will  add  unnecessary  delay  and  cost  to  the  project.  Likewise,  if  high  or 
medium-risk  projects  are  categorized  and  treated  incorrectly,  they  will  not  be  processed 
correctly.  Potentially,  additional  cost  and  time  will  be  incurred  because  these  projects 
have  not  been  tested  and  evaluated  properly,  or  had  other  program  risks  that  have  not 
been  properly  addressed.  Acquisition  strategies  should  also  be  updated  and  resubmitted 
promptly  when  conditions  change  significantly.  This  will  allow  decision-makers  to 
properly  assess  the  impact  of  such  changes.  Users  of  the  items  should  be  involved  early 
in  the  acquisition  process  in  order  to  fully  understand  the  nature,  potential  limitations, 
and  advantages  of  commercial  and  nondevelopmental  items.  Only  in  this  way  can  the 
user  and  acquisition  community  maintain  the  flexibility  that  is  needed  in  establishing 
requirements  to  make  effective  use  of  commercial  items  and  nondevelopmental  items  in 
their  programs. 

2.  Market  Investigations 

Current  DoD  policy  encourages  the  use  of  commercial  products.  However, 
procurement  agencies  have  the  burden  of  determining  whether  commercial  items  are 
available.  This  process  is  called  market  research. 

Market  research  consists  of  two  related  techniques:  market  surveillance  and 
market  investigation.  Market  surveillance  is  used  to  maintain  a  current  knowledge  of 

market  availability  within  a  particular  area  of  technical  expertise.  Market  investigations 
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evaluate  eommereial  items  in  the  marketplaee  to  determine  whether  they  ean  meet 
speeifie  military  requirements,  or  whether  a  new  aequisition  development  projeet  should 
be  initiated. 

Market  researeh  is  defined  in  more  detail  in  ehapter  II,  paragraph  E,  of  this  thesis. 
Market  investigations  should  satisfy  three  eriteria  as  follows:  (1)  identify  all  potential 
eommereial  item  or  nondevelopmental  item  sourees  (whieh  ineludes  other  military 
serviees,  other  governmental  ageneies  and  allied  eountries),  (2)  be  eoordinated  with 
independent  evaluation  aetivities,  and  (3)  obtain  enough  aetual  test  and  performanee  data, 
if  applieable,  to  assess  eritieal  issues  relating  to  the  operational  suitability  of  the 
eommereial  or  nondevelopmental  item. 

3,  Concept  Formulation 

The  materiel  developers  (U.S.  Army  Materiel  Command)  and  the  eombat 
developers  (U.S.  Army  Training  and  Doetrine  Command)  jointly  prepare  the  eoneept 
formulation  paekage  in  the  Army  aequisition  proeess.  The  eoneept  paekage  ineludes  a 
tradeoff  determination  and  a  best  teehnieal  approaeh,  whieh  are  prepared  by  the  material 
developers.  It  also  ineludes  an  Analysis  of  Alternatives,  whieh  is  eompleted  by  the 
eombat  developers.  The  eoneept  formulation  proeess  provides  the  initial  up-front  and 
early  data  for  defining  and  approving  the  material  requirements  doeuments  and 
supporting  program  deeisions,  ineluding  analysis  of  aequisition  strategies.  A  joint  proeess 
is  used  to  ensure  that  the  material  requirements  are  realistie,  affordable  and  analytically 
supported.  These  analyses,  along  with  the  market  investigation,  form  the  basis  for 
development  of  a  sound  acquisition  strategy  and  formal  requirements  document. 

4,  Management  of  Integration 

Integration  projects  -  that  is,  projects  that  may  use  commercial  items  but  have  to 
integrate  components  or  systems  to  work  together  -  can  be  high-risk,  complex  and  costly 
acquisition  efforts  and  therefore  must  be  managed  appropriately.  If  integration  projects 
are  not  identified  properly,  there  will  be  high  risk  of  a  program  experiencing  technical 
problems  and  not  meeting  user  requirements.  Integration  projects  usually  present  the 
greatest  challenges  and  are  among  the  highest  risk  programs  that  utilize  commercial 
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items.  The  adverse  results  eould  impaet  eosts  and  schedules  as  well  as  not  meeting  user 
requirements,  resulting  in  potential  mission  failure. 

5.  Logistics  Support 

Logistics  support  planning  is  a  process  that  should  commence  early  in  a  new 
acquisition  program.  The  goal  is  to  influence  material  system  requirements  and  design  in 
order  to  ensure  that  the  system  has  optimum  reliability  and  life  cycle  costs  (LCC)  -  in 
other  words,  maximum  reliability  and  minimum  life  cycle  costs.  Poor  logistics  support 
planning  could  result  in  inadequate  logistics  support,  which  in  turn  could  adversely  affect 
the  system’s  readiness  and  cost. 

a.  Reliability 

Reliability  is  defined  as  the  “probability  that,  when  operating  under  stated 
conditions,  the  system  or  component  will  perform  its  intended  function  adequately  for  a 
specified  period  of  time.”  [Air  Force  Space  and  Missile  Command,  pg.  1,  14  August 
1998] 

In  programs  involving  R&D,  a  major  issue  that  developers  must  consider 
is  system  reliability.  Reliability,  availability,  and  maintainability  (RAM)  are  key  LCC 
drivers.  RAM  must  be  addressed  early  in  the  acquisition  process  by  establishing 
reliability  requirements  that  determine  whether  the  system  is  economically  supportable 
over  its  entire  life  cycle  and  is  able  to  achieve  required  readiness  levels. 

If  the  reliability  of  a  commercial  item  is  not  verified  as  being  able  to  meet 
the  user’s  requirements,  then  the  system’s  LCC  and  readiness  will  be  severely  impacted. 
Lower  reliability  greatly  impacts  the  support  costs,  system  availability  and  thus,  mission 
accomplishment.  RAM  requirements  must  be  established  early  in  order  to  enable  the 
adequate  testing  of  commercial  items  or  nondevelopmental  items.  Quantitative  reliability, 
availability  and  maintainability  requirements  should  be  established  for  all  commercial 
items,  with  the  exception  of  passive  commercial  items  with  no  identifiable  failures. 

b.  Support  Requirements 

Support  requirements  have  to  do  with  the  way  the  system  is  maintained.  It 
can  be  supported  by  either  a  contractor  supply  and  maintenance  system  or  a  military 
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organic  (in-house)  supply  and  maintenance  system.  Contractor  or  organic  support 
requirements  must  be  analyzed  via  cost-benefit  analysis  methodology  in  order  to  justify 
the  selected  support  decision.  Support  requirements  specifically  address  how  commercial 
items  will  be  maintained  and  supported  when  fielded  due  to  the  fact  that  Government 
access  to  commercial  item  technical  data  will  most  likely  be  limited.  Consequently, 
availability  of  supplied  parts  may  cease  before  the  intended  military  lifecycles  of  the 
items  expire.  Support  requirements  analysis  is  crucial,  since  military  systems  are  kept  in 
the  operational  inventory  far  longer  then  the  lifespan  of  most  commercial  items, 
c.  Technical  Data  for  Competitive  Reprocurement 

Technical  data  comes  in  many  forms,  including  technical  maintenance  and 
operating  manuals,  engineering  drawings  and  specifications,  repair  parts  lists,  and  special 
tool  lists,  to  name  a  few.  All  acquisition  programs  require  technical  data  to  some  degree 
for  logistics  support  of  the  system.  The  data  that  is  acquired  for  a  new  program  should  be 
tailored  to  the  specific  and  well  planned  needs  of  the  program.  Generally,  the  need  for 
technical  data  to  support  reprocurement  decisions  should  be  made  via  economic  analysis, 
which  will  indicate  the  best  overall  strategy  for  obtaining  spare  parts  or  replacement 
systems.  As  previously  mentioned.  Government  access  to  commercial  item  or 
nondevelopmental  item  technical  data  will  most  likely  be  limited,  and  availability  of  parts 
supplied  may  cease  before  the  intended  military  lifecycles  of  the  items  expire.  An 
analysis  of  the  program’s  future  requirements  for  technical  data  is  essential,  since 
military  systems  are  kept  in  the  operational  inventory  far  longer  then  the  lifespan  of  most 
commercial  items.  Thus,  there  must  be  some  provisions  made  for  the  supply  of  spare 
parts  after  the  commercial  items  have  become  obsolete  in  the  commercial  marketplace. 

6.  Test  and  Evaluation 

The  purposes  of  test  and  evaluation  in  the  development  of  a  defense  system  and 
its  acquisition  process  are  the  following:  to  determine  the  feasibility  of  conceptual 
approaches  in  order  to  minimize  design  risk,  to  identify  design  alternatives,  compare  and 
analyze  tradeoffs,  and  to  estimate  operational  effectiveness  and  suitability.  As  a  system 
progresses  through  design  and  development,  the  emphasis  in  testing  moves  gradually 
from  development  test  and  evaluation  (DT&E)  -  which  is  chiefly  concerned  with  the 
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attainment  of  engineering  design  goals  -  to  operational  test  and  evaluation  (OT&E), 
whieh  foeuses  on  questions  of  operational  effeetiveness,  suitability,  and  supportability. 

DT&E  is  normally  planned  and  eondueted  by  the  developing  ageney  during  the 
eoneept  and  teehnology  development  phase  to  assist  in  the  seleetion  of  the  preferred 
alternative  system  eoneepts,  teehnologies,  and  designs.  During  the  system  development 
and  demonstration  phase,  DT&E  demonstrates  that  engineering  is  reasonably  eomplete, 
that  all  signifieant  design  problems  are  in  hand,  and  that  the  design  meets  its  required 
speeifieations  in  all  areas,  such  as  performance,  reliability,  and  maintainability. 

The  purpose  of  OT&E  is  to  verify  that  only  operationally  effective  and 
operationally  suitable  systems  are  approved  for  production,  thereby  meeting  both  mission 
needs  and  the  minimum  operational  performance  requirements  of  the  user.  Eor  major 
systems,  OT&E  is  normally  planned  and  conducted  by  a  major  OT&E  field  agency 
located  within  the  DoD  component.  This  operational  test  agency  must  be  separate  and 
independent  from  the  developing/procuring  agency. 

Operational  testing  is  the  primary  means  of  evaluating  system  performance  in  a 
combat  representative  environment.  It  is  structured  to  determine  the  operational 
effectiveness  and  suitability  of  a  system,  as  well  as  to  determine  if  the  minimum 
acceptable  operational  performance  requirements  have  been  satisfied. 

Operational  effectiveness  refers  to  the  ability  of  a  system  to  accomplish  its 
mission  in  the  planned  operational  environment.  Suitability  is  the  degree  to  which  a 
system  can  be  placed  satisfactorily  in  field  use,  taking  into  account  such  factors  as  the 
system’s  reliability  and  maintainability.  [DSMC,  January  1990] 

In  the  commercial  arena,  commercial  items  may  already  have  test  data  available; 
however,  unless  the  data  has  been  verified  by  an  independent  evaluator  it  should  not  be 
accepted. 

In  all  usage  of  commercial  items  in  major  programs,  OT  must  be  performed  in 
order  to  verify  or  establish  the  system’s  performance  in  the  representative  operational 
wartime  environment. 
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7. 


Culture 


It  has  been  found  that  culture  plays  a  key  role  in  many  negative  decisions  that  are 
made  in  the  acquisition  community.  As  used  here,  culture  can  be  defined  as  the 
background,  mores,  traditions  and  customs,  or  way  of  life  of  a  group. 

In  the  acquisition  arena,  there  are  many  key  players  and  contributors  to  the  PM’s 
mores,  traditions,  and  way  of  life.  Some  of  the  key  players  are  the  military  chain-of- 
command,  the  Congressional  and  military  budgeters,  the  program’s  contractor,  the 
Government  contracting  agency,  the  program  decision  authority,  the  user  community, 
and  the  test  community,  all  with  needs  that  may  diverge  and  actually  threaten  program 
success. 

For  example,  there  are  Congressional  budget  “rewards”  that  force  a  PM  to  do 
everything  within  his  power  to  see  that  his  program  doesn’t  fail  a  test.  If  he  fails  a  test,  he 
may  not  have  a  program,  because  his  program  funding  will  be  cut.  Therefore,  he  doesn’t 
have  a  keen  interest  in  failing  as  many  times  as  is  necessary  to  adequately  “learn”  from  a 
developmental  test.  Maturing  the  technology  through  the  test-fix-test  process  would 
ensure  that  his  program  will  not  have  later  technological  setbacks  that  would  in  turn  drive 
schedule  and  cost  overruns,  as  well  as  increasing  life-cycle  support  costs  of  the  program. 
Nor  does  the  PM  welcome  the  tester,  when  he  says,  “I’m  here  to  help  you.”  The  culture 
and  reward  system  by  which  the  PM  lives  dictates  a  different  response.  As  testimony  to 
this,  a  GAO  study  found  that  the  processes  and  culture  within  DoD  have  contributed  to  a 
feeling  of  great  aversion  by  PMs,  both  towards  the  test  community  as  well  as  towards  the 
testing  of  the  PMs’  weapon  systems.  The  GAO  states  the  following: 


Testing  plays  a  less  constructive  role  in  DoD  because  a  failure  in  a  key  test 
can  jeopardize  program  support.  Specifically,  test  results  often  become 
directly  linked  to  funding  and  other  key  decisions  for  programs.  Such  a 
role  creates  a  more  adversarial  relationship  between  testers  and  program 
managers.  [GAO,  pg.  6,  31  July  2000] 
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The  negative  cultural  pressures  on  a  PM  are  further  exemplified  by  an  article  in 
the  Marine  Corps  Times,  titled,  “We  Need  to  Lie  —  To  Save  the  Osprey,  Squadron  Boss 
Ordered  Marines  to  Fake  Records.”  [Brinkley,  pg.8,  29  January  2001]  This  article 
alludes  to  the  negative  cultural  pressures  as  perceived  by  the  Marine  PM,  Lt.  COL.  Odin 
F.  Liberman  that  forced  him  to  lie  to  protect  his  program.  The  article  was  based  upon  an 
anonymous  letter  and  audiotape  received  by  the  administrative  offices  of  Navy  Secretary 
Richard  Danzig.  The  article  quotes  the  barely  audible  audiotape  of  a  voice  identified  as 
Liberman’s  that  states. 


The  reason  we  need  to  lie  is  . . .  [to]  manipulate  the  data  until  Milestone  III 
when  the  aircraft  passes  its  final  evaluation  and  enters  full  rate  production, 
this  program  is  in  jeopardy  ...  Everybody  says  readiness  is  in  bad  ...  It’s 
something  that  everybody  is  focused  upon. 

[Brinkley,  pgs  8  &  10,  29  January  2001] 


8,  Army  Definitions 

During  the  time  of  the  Army  audits  cited  below,  the  definition  of  the  various 
classifications  of  what  was  known  then  as  nondevelopmental  items  (which  would  be 
classified  now  as  commercial  items),  is  quoted  as  follows: 


•  Basic.  These  are  off-the-shelf  items  used  in  the  same  environment  for 
which  they  were  designed.  The  items  should  not  require  development  or 
modification  to  be  acceptable.  Available  information  (such  as  test  data, 
technical  publications  and  reliability  data)  should  be  obtained  from  the 
item  source  and  evaluated  to  avoid  costly  duplication  of  effort. 

•  Adaptation.  These  are  off-the-shelf  items  adapted  for  use  in  a  different 
environment  that  they  were  designed  for.  The  items  require  some 
modification  for  military  use.  Test  and  evaluation  usually  is  required  to 
ensure  the  modified  items  meet  user  requirements. 

•  Integration,  These  are  nondevelopmental  items  used  as  subsystems  or 
components  of  larger  systems.  The  efforts  require  a  prove-out  phase 
involving  systems  engineering  and  test  and  evaluation  to  make  sure  the 
total  system  meets  user  requirements  and  is  producible. 

[USAAA,  pg.  8,  19  March  1990] 
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c. 


COMMERCIAL  ITEM  USAGE  ISSUES 


Issues  will  be  presented  by  audit  ageney  report  or  issue  paper  topie  and  then 
segregated  into  sueh  areas  as  needs  determination,  aequisition  strategy,  market 
investigation,  eoneept  formulation,  design,  risk  analysis,  management  of  integration,  test 
and  evaluation,  and  logisties  support.  Logisties  support  will  be  further  divided  into  the 
following  areas;  reliability,  support  requirements,  and  teehnieal  data  for  eompetitive 
reprocurement. 

1.  U,S,  Army  Audit  Agency,  NE  90-206,  Acquisition  of 
Nondevelopmental  Items:  U.S.  Army  Natick  Research,  Developmental  and 
Engineering  Center,  Natick,  Massachusetts,  19  March  1990 

This  audit  reports  that  the  Army  often  based  acquisition  strategies  on  incomplete 

information  due  to  the  fact  that  the  Army  did  not  properly  conduct  market  investigations 
and  concept  formulation  analysis. 

Some  of  the  Army  Audit  Agency  specific  findings  were  as  follows:  “One  system 
. . .  involved  system  integration,  but  went  from  program  initiation  to  production  without 
sufficient  testing.”  [Pg  2]  This  resulted  in  an  Army  worldwide  modular  print  system 
being  purchased  without  sufficient  required  testing,  which  would  have  ensured  that  the 
system  met  Army  requirements.  “Planned  maintenance  concept  for  an  automatic  reserve 
parachute  ripcord  release  was  inconsistent  with  requirements  documents,  and  . . .  did  not 
tell  decision  makers  the  true  cost  or  extent  of  required  overhauls.”  [pg.  2] 
a.  Acquisition  Strategy 

The  Army  Audit  Agency  reviewed  15  of  22  commercial  item  projects,  and 
found  that  14  of  the  15  projects  did  not  identify  the  Army  identification  category  of 
commercial  item  (i.e.,  basic  off-the-shelf,  adaptation  or  integration  item).  For  10  of  these 
projects,  the  strategy  was  inconsistent  with  the  type  of  category  of  commercial  item  being 
acquired.  For  example,  basic  off-the-shelf  items  were  managed  as  adaptation  projects.  In 
turn,  adaptation  and  integration  items  were  managed  as  other  types  of  projects.  For  basic, 
low -risk,  off-the-shelf  projects,  this  added  unnecessary  delay.  In  addition,  complex,  high- 
risk  integration  projects  that  should  have  been  rigorously  tested  and  evaluated  were 
instead  rushed  into  production  as  if  they  had  been  off-the-shelf  projects.  For  example,  as 
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the  Army  Audit  Agency  states  “One  system  —  the  Modular  Print  System  —  involved 
system  integration,  but  went  from  program  initiation  to  production  without  sufficient 
testing.”  [pg.  2]  This  project  was  actually  a  complex  system  integration  effort  involving 
risks. 

These  problems  resulted  from  the  erroneous  supposition  that  by 
eliminating  the  necessary  required  analysis  developers  would  be  able  to  streamline  the 
acquisition  process.  As  the  Army  Audit  Agency  report  says,  “Natick  Center  personnel 
did  not  appear  to  fully  understand  the  various  categories  of  commercial  items  and  did  not 
believe  the  category  was  important.”  [pg.  11] 

b.  Market  Investigations 

The  Army  Audit  Agency  found  that  market  investigations  either  were  not 
conducted  at  all,  or  were  inadequate  for  most  of  the  commercial  item  projects.  Army 
Audit  Agency  reviewed  15  of  22  commercial  item  projects,  and  found  that  five  market 
investigations  were  not  documented.  Furthermore,  for  seven  of  the  projects,  the  market 
investigations  were  neither  complete,  nor  had  been  coordinated  with  the  independent 
evaluators  as  required.  The  Natick  RD&E  Center’s  policy  at  the  time  was  to  leave  market 
investigations  incomplete  or  undocumented  in  order  to  streamline  the  acquisition  process, 
thereby  saving  time.  However,  without  properly  conducted  market  investigations, 
managers  had  little  assurance  that  all  the  possible  alternatives  or  material  sources  had 
been  identified,  investigated,  and  that  the  best  alternative  had  been  selected.  The  Army 
Audit  Agency  said,  “Natick  Center  often  did  not  properly  conduct  market  investigations 
and  other  required  analyses  to  develop  sound  acquisition  strategies  for  nondevelopmental 
items.”  [pg.  2] 

For  example,  both  snow  and  ice-traversing  equipment  and  a  field  pack 
project  experienced  costly  problems  by  not  having  a  market  investigation  completed.  In 
the  case  of  the  snow  and  ice-traversing  equipment,  if  a  proper  market  investigation  had 
been  performed,  the  problems  with  the  selected  off-the-shelf  items  could  have  been 
recognized  and  evaluated  prior  to  acquiring  and  testing  them.  In  the  case  of  the  field 
pack,  a  market  investigation  would  have  identified  the  deficiencies  with  the  commercial 
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pack,  which  required  modifieation  to  sueh  an  extent  that  it  bore  no  resemblanee  to  the 
original  eommercial  paek. 

c.  Concept  Formulation 

The  research  center  had  not  prepared  eoneept  formulation  paekages  for 
any  of  the  15  projeets  reviewed,  resulting  in  many  questionable  deeisions.  For  example, 
the  Army  developed  a  saltwater  purification  device  at  a  eost  of  $250,000  and  a  purehase 
eost  of  $1,320  eaeh,  when  there  was  already  a  Navy-developed  system  available  for  a 
purehase  eost  of  $980  eaeh.  A  proper  eoneept  formulation  paekage  would  have  identified 
the  Navy  system  as  a  possible  alternative.  A  tradeoff  analysis  between  eost  and  differing 
requirements  eould  have  resulted  in  seleetion  of  the  Navy  system  as  an  alternative,  thus 
saving  the  Army  $250,000  in  development  eosts  as  well  as  the  extra  $340  purehase  eost 
per  system.  This  seleetion  would  also  have  saved  the  Army  the  two  years  of 
developmental  time  needed  to  obtain  the  system. 

Another  example,  with  no  eoneept  formulation  paekage,  is  a  five-soldier 
erew  tent  projeet,  whieh  resulted  in  five  eommereial  tents  being  purehased  and  evaluated 
three  times  at  a  eost  of  $600,000.  A  proper  eoneept  formulation  paekage,  ineluding 
tradeoff  and  other  analyses,  would  have  identified  the  most  promising  alternative  for 
decision  makers. 

2.  U,S,  Army  Audit  Agency,  NE  91-204,  Acquisition  of 

Nondevelopmental  Items.  17  June  1991 

The  audit,  among  other  findings,  found  that  five  separate  Army  eommodity 
eommand  groups  managing  39  eommereial  item  projeets  didn’t  properly  support 
aequisition  strategies. 

a.  Acquisition  Strategy 

The  analysis  to  support  the  seleetion  of  eommereial  items,  whieh  is 
necessary  at  the  beginning  of  a  projeet,  was  not  performed.  As  the  Army  Audit  Ageney 
found,  “DA  (Department  of  the  Army)  sometimes  direeted  developers  to  use 
nondevelopmental  items,  and  managers  therefore  believed  there  was  no  real  need  to 
perform  the  eomprehensive,  upfront  analysis  required  to  support  their  strategies.”  [pg.  19] 
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b.  Market  Investigations 

Market  investigations  were  not  documented  for  11  of  the  39  projects.  Of 
the  remaining  28  projects,  a  total  of  21  had  incomplete  documentation.  As  the  Army 
Audit  Agency  noted,  “The  investigations  didn’t  adequately  support  the  streamlining 
actions  managers  took — particularly  the  elimination  of  a  development  or  prototype  phase 
before  production  ...  As  a  result,  the  development  effort  required  to  adapt  or  integrate 
these  items  had  to  occur  during  production.”  [pg.  25] 

c.  Concept  Formulation 

Of  29  projects  reviewed  for  a  complete  concept  formulation  package,  none 
supported  this  concept  formulation  characteristic.  The  majority,  22  projects,  didn’t  have 
any  of  the  concept  package  completed.  For  the  7  projects,  only  the  portion  of  the  concept 
formulation  package  that  dealt  with  cost  and  operational  effectiveness  was  complete. 

d.  Management  of  Integration 

As  the  Army  Audit  Agency  stated  in  this  report  regarding  integration 
efforts,  “Such  efforts  not  only  require  research  and  development,  but  may  need  to 
undergo  both  a  demonstration  and  full-scale  development  phase  unless  risk  is  determined 
to  be  sufficiently  low.”  [pg.  28]  Eight  of  the  Army  potentially  high-risk  integration 
projects,  totaling  $1.7  billion,  didn’t  have  a  research  and  development  (R&D)  phase  prior 
to  production.  Although  utilizing  commercial  item  off-the-shelf  components,  the  systems 
were  high-risk  integration  projects  for  which  the  Army  had  documented  only  minimal 
information  to  show  that  the  projects  would  meet  user  requirements  after  integration  was 
accomplished. 

e.  Logistics  Support 

The  Army  Audit  Agency  reviewed  39  commercial  item  projects  for  which 
logistics  support  costs  had  been  projected  at  $24  billion.  These  projects  had  critical  issues 
related  to  logistics  support,  which  hadn’t  been  addressed  early  enough  in  the  acquisition 
process.  Three  of  these  issues  are  as  follows:  reliability,  support  requirements,  and 
technical  data  for  competitive  reprocurement. 
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(1)  Reliability.  The  Army  Audit  Ageney  reviewed  39 
eommereial  item  programs,  many  with  major  defieiencies.  They  found  that  11  of  these 
projeets  had  no  quantitative  reliability  requirements  established.  In  13  of  the  projects, 
reliability  had  not  been  adequately  addressed  during  market  investigations.  In  several 
cases,  contractors’  reliability  estimates  had  been  received  without  the  performance  data, 
which  is  necessary  in  order  to  evaluate  the  contractor  reliability  claims.  In  15  of  the 
reviewed  projects,  the  Army  Audit  Agency  found  that  material  developers  hadn’t 
adequately  conducted  reliability  testing  prior  to  award  of  the  production  contract.  It  was 
found  that  systems  were  fielded  that  had  subsequent  reliability  problems,  which  could 
affect  system  readiness  and  significantly  increase  costs.  As  the  Army  Audit  Agency  said, 
“The  failure  to  adequately  address  reliability  early  in  the  acquisition  process  for  new 
programs  significantly  increases  the  risks  associated  with  using  nondevelopmental  items 
to  satisfy  Army  requirements.”  [pg.  40] 

(2)  Support  Requirements.  Many  of  the  39  projects 
reviewed  had  no  reliability  testing,  which  would  have  reduced  the  risk  of  critical 
decisions  about  logistics  support.  In  all  cases,  the  most  expeditious  means  available  were 
used  to  establish  the  logistic  support  plans,  which  were  not  necessarily  the  best  or  most 
cost-effective  decisions  for  the  Army. 

(3)  Technical  Data  for  Competitive  Reprocurement.  For  14 
of  the  39  projects  reviewed,  data  adequate  for  reprocurement  was  not  acquired. 
Consequently,  the  Army  will  have  to  use  the  sole  source  vendor  for  any  reprocurement  of 
spare  parts  or  systems  throughout  their  complete  lifespan.  Some  managers  assessed  the 
availability  of  data  suitable  for  competitive  reprocurement  in  their  market  analysis.  But 
many  other  market  investigations  either  were  not  documented,  or  the  availability  of 
technical  data  was  not  addressed  during  the  market  investigations.  Some  managers  used 
lifetime  contractor  support,  not  having  considered  technical  reprocurement  data  as 
necessary.  Others  decided  that  data  necessary  for  reprocurement  was  too  expensive.  In 
most  cases,  managers  didn’t  use  economic  analysis  or  any  other  proper  analytical 
technique  prior  to  their  decision.  It  was  determined  by  the  Army  Audit  Agency  that 
several  projects  would  have  benefited  from  obtaining  technical  data  suitable  for 
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competitive  reproeurement.  Those  programs  either  had  to  aequire  replenishment  spares 
at  signifieantly  higher  priees  (Lightweight  Digital  Faesimile),  or  had  to  find  a 
replaeement  unit  beeause  spares  and  eontraetor  support  eould  not  be  obtained  (Basie 
Generation  Unit  used  in  the  Battlefield  Eleetronies  Communieation  System),  [pgs.  44-45] 

3.  U,S,  Army  Audit  Agency,  NE  90-209,  Acquisition  of 

Nondevelopmental  Items:  Fort  Monmouth,  New  Jersey.  11  September  1990 

This  audit  found  that  the  Army  advoeate  for  eommereial  items  had  not  been 
involved  enough  to  make  sure  that  program  managers  eonsidered  the  eommereial  item 
approaeh.  Furthermore,  it  was  diseovered  that  the  advoeates  were  not  adequately 
monitoring  eommereial  item  projeets  to  doeument  “lessons  learned,”  or  to  make  sure  that 
the  two  main  objeetives  of  eommereial  items  were  aehieved:  the  reduetion  of  program 
eost  and  aequisition  time.  Also,  market  investigations  and  eoneept  formulation  analyses 
either  were  not  eondueted  at  all  or  were  improperly  eondueted,  and  projeet 
doeumentation  was  either  lax  or  nonexistent.  Complex  integration  projeets  were  awarded 
without  adequate  testing  to  assure  that  they  were  operationally  suitable  and  that  they  met 
the  Army  requirements.  Support  eoneepts  were  not  adequately  investigated  or  justified  to 
demonstrate  the  eost-effeetiveness  of  the  seleeted  support  eoneept  during  daily  use, 
espeeially  under  wartime  eonditions. 

a.  Acquisition  Strategy 

Key  phases  and  eritieal  steps  of  the  aequisition  proeess  were  eliminated 
without  explanation  for  all  of  the  15  eommereial  item  projeets  reviewed.  Thus, 
aequisition  strategies  that  were  followed  were  not  adequately  supported  by  analysis.  As  a 
result,  managers  often  followed  inappropriate  strategies  for  the  eategory  of  eommereial 
item  aequired.  For  example,  eomplex  eommereial  items  requiring  integration  were 
frequently  aequired  with  a  strategy  more  appropriate  for  low-risk,  off-the-shelf  items. 
Seven  eomplex  eommereial  item  projeets  went  into  produetion  without  a  separate  R&D 
phase  to  obtain  and  evaluate  prototypes.  When  any  phase  of  the  aequisition  proeess  is 
eliminated  or  eombined.  Army  regulation  requires  that  the  material  developer  justify  their 
deeision  in  the  aequisition  strategy.  Most  of  the  strategies  reviewed  simply  stated  that  the 
eommereial  item  approaeh  ehosen  eliminated  the  need  for  researeh  and  development. 
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b.  Market  Investigations 

Market  investigations  were  not  doeumented  for  five  of  the  15  eommercial 
item  projects  reviewed.  There  was  documentation  to  show  that  market  investigations  had 
been  started  for  10  of  the  15  projects;  however,  they  were  incomplete  and  unacceptable 
for  decision-making  purposes.  In  some  cases,  PMs  cited  “urgent  requirement”  as  the 
reason  for  inadequate  market  investigation.  However,  even  follow-on  contract  options 
were  awarded  without  justification  for  not  having  a  market  investigation. 

c.  Concept  Formulation 

The  concept  formulation  process  was  not  completed  for  any  of  the  15 
projects  reviewed.  The  acquisition  strategies  did  not  address  this  key  step,  nor  did  they 
explain  why  the  required  concept  formulation  process  was  not  performed.  The  Army 
Audit  Agency  found  that  PMs  felt  that  since  they  had  been  directed  to  use  commercial 
items  by  higher  management,  there  was  neither  a  need  nor  a  requirement  for  the  concept 
formulation  phase. 

d.  Test  and  Evaluation 

Production  contracts  for  12  of  the  15  projects  were  begun  without 
independent  test  and  evaluation  to  determine  whether  the  items  met  Army  requirements. 
These  contracts  were  initiated  in  spite  of  the  fact  that  all  categories  of  commercial  items 
are  required  to  undergo  operational  test  and  evaluation  (OT&E)  before  production,  unless 
an  independent  market  investigation  report  concludes  that  such  testing  can  be  reduced  or 
eliminated. 

During  a  review  of  15  commercial  item  projects,  the  Army  Audit  Agency 
also  found  that  1 1  items  did  not  have  OT  performed.  As  the  Army  Audit  Agency  says, 
“In  total,  about  $1.8  billion  was  spent  acquiring  nondevelop mental  items  without 
adequately  showing  the  suitability  of  the  items  in  a  realistic  operating  environment.” 

[pg.31] 
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e.  Logistics  Support 

An  Army  Audit  Agency  review  of  ten  commercial  item  projects,  with 
estimated  LCC  support  costs  of  $24  billion,  had  no  analysis  performed  that  justified  the 
logistics  support  concept  selected. 

In  the  AN/PRC- 127  Radio,  for  example,  the  determination  was  made  to 
use  a  throwaway  repair  concept  to  support  the  radio’s  receiver-transmitter  after  the 
manufacturer’s  warranty  expired.  A  maintenance  cost-effectiveness  analysis  performed 
prior  to  contract  award  concluded  that  repair  costs  were  about  the  same  as  replacement 
costs,  hence  the  throwaway  decision.  However,  when  the  contract  was  awarded,  the 
actual  unit  price  was  much  higher  than  maintaining  the  receiver-transmitter  would  have 
been.  Nonetheless,  the  support  concept  was  not  reevaluated. 

(1)  Reliability.  The  Army  Audit  Agency  reviewed  seven 
commercial  item  projects  for  their  reliability.  They  found  that  the  material  developers  did 
not  ensure  that  qualitative  reliability  requirements  were  documented  during  the  concept 
exploration  phase,  prior  to  awarding  the  production  contracts.  Three  of  the  commercial 
item  programs’  requirements  were  not  established  at  all.  In  the  other  four  programs, 
managers  used  the  contractor’s  predicted  values  for  the  quantitative  requirements,  which 
were  basically  technical  estimates  and  not  based  on  analysis  of  the  item’s  operating 
mission. 

(2)  Supportability.  An  Army  Audit  Agency  review  of  ten 
commercial  item  projects  with  estimated  life  cycle  support  costs  of  $24  billion  had  no 
analysis  performed  that  justified  the  logistics  support  concept  selected.  Logistics  support 
plans  were  usually  prepared,  but  there  was  no  analysis  to  justify  those  plans  or  to 
demonstrate  cost-effectiveness  of  the  support  concepts.  In  most  cases,  the  end-item 
contractor  was  to  provide  maintenance  and  supply  support  for  the  life  expectancy  of  most 
items.  However,  there  was  no  analysis  performed  that  showed  that  contractor  support 
was  more  cost  effective  than  using  the  Army’s  organic  support  system.  For  example,  as 
the  Army  Audit  Agency  indicated,  “Program  plans  did  not  ensure  that  spare  parts  for  the 
AN/PRC-126  Small  Unit  Radio  would  be  available  at  fair  and  reasonable  prices.”  [pg.  5] 
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4,  Bonheim,  Mike  LTC,  PM  -  Physical  Security  Equipment  (PSE), 
Commercial  Items  Acquisition  Data  Call,  22  February  2001 

PM-PSE  acts  as  the  PSE  focal  point  for  the  Army  and  other  Services.  The  DoD 
Physical  Security  Equipment  Action  Group  has  created  a  Commercial  Off-The-Shelf 
working  group  to  identify  commercial  physical  security  and  force  protection  equipment 
that  can  immediately  be  used  to  solve  terrorism  deficiencies.  Under  the  Chairman  Joint 
Chiefs  of  Staff  Instruction,  CJCSI  5261. 01 A,  commanders  are  provided  funds  to  react  as 
follows:  “The  fund  provides  a  means  for  CINCs  to  react  to  unanticipated  requirements 
from  changes  in  terrorist  threat  level  or  force  protection  doctrine/  standards.”  [CJCS,  pg. 
1] 

Therefore,  as  the  PSE  focal  point,  PM-PSE  has  been  involved  in  commercial  item 
security  equipment  projects,  identifying  several  problems  as  follows: 

a.  Needs  Determination 

A  variety  of  Government  users  are  involved  in  the  purchase  of  security 
equipment.  Eor  the  most  part  these  users  have  no  experience  or  training  in  the  acquisition 
process.  This  results  in  difficulty  in  obtaining  a  user  statement  of  work  that  accurately 
identifies  what  they  need  or  want  the  product  to  do. 

b.  Reliability 

There  may  be  a  multitude  of  buyers  from  different  commands  for  the 
security  equipment,  each  having  their  own  certification  requirements.  Consequently,  a 
vendor  will  not  be  able  to  certify  the  product  for  all  scenarios. 

Also,  the  vendors’  performance  claims  have  proven  inaccurate.  As  PM- 
PSE  states,  “In  cases  involving  two  portable  barriers  both  barriers  failed  the  initial  crash 
test  and  had  to  be  reconfigured  by  the  manufacturer  before  a  successful  test  could  be 
accomplished.”  [Bonhiem,  pg.  2] 

c.  Market  Investigations 

There  has  been  no  centralized  database  established  for  equipment  that  has 
been  purchased  as  well  as  the  results  of  their  operational  use  or  testing. 
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d.  Logistics  Support 

There  are  no  durability  or  lifespan  indicators  for  these  physical  security 
commercial  items.  Repair  procedures  are  nonexistent.  PM-PSE  states,  “In  many  cases, 
detailed  repair  documentation  is  not  available.  If  the  item  fails  it  must  be  scrapped  or 
returned  to  the  factory  for  repair  -  a  much  more  difficult  task  in  deployed  units  than  the 
civilian  world.”  [Bonhiem,  pg  2]  Determining  the  quantity  and  type  of  spare  parts  is  an 
issue.  As  PM-PSE  indicates,  “Determining  the  amount  of  repair  and  spare  parts  and 
where  there  are  stored  and  subsequently  issued  needs  to  be  determined.”  [Bonhiem,  pg. 
2] 

e.  Test  and  Evaluation 

There  may  be  a  multitude  of  buyers  from  different  commands  for  the 
security  equipment,  each  with  their  own  testing  requirements.  Consequently,  a  vendor 
will  not  be  able  to  test  or  certify  the  product  for  all  scenarios. 

5.  GAO/NSIAD-95-161,  UNMANNED  AERIAL  VEHICLES:  Maneuver 

System  Schedule  Includes  Unnecessary  Risk,  15  September  1995 

The  GAO  states  that  because  of  premature  entry  into  production,  two  previous 
UAV  programs  suffered  from  costly  redesigns  in  order  to  achieve  acceptable  system 
performance.  The  two  previous  nondevelopmental  UAV  systems,  the  Pioneer  and  the 
Hunter,  were  both  started  into  production  without  being  subjected  to  any  operational 
testing. 

Premature  production  of  the  Pioneer  resulted  in  doubling  the  costs  for  nine 
systems  that  did  not  meet  the  operational  requirements  of  the  user.  As  of  the  writing  of 
this  GAO  report,  the  Hunter  system  was  also  experiencing  problems  and  an  uncertain 
future  after  an  expenditure  of  $627  million. 

Because  of  the  previous  dubious  acquisition  track  record,  the  GAO  issued  this 
interim  report  to  bring  attention  to  inadequate  testing  aspects  of  the  program  that,  it  was 
believed,  would  unnecessarily  increase  DoD’s  program  risk. 
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Test  and  Evaluation 


This  report  states  that  the  Joint  Taetieal  Unmanned  Aerial  Vehicles  project 
office  planned  to  begin  production  of  the  Maneuver  System  before  making  sure  that  the 
system  could  meet  operational  requirements  via  operational  testing  prior  to,  or  during, 
low  rate  initial  production  (TRIP).  Test  articles  are  normally  obtained  from  TRIP; 
however,  in  this  case  there  was  no  testing  of  the  TRIP  vehicles  planned  by  the  project 
office. 

6,  GAO/NSIAD-96-2,  Unmanned  Aerial  Vehicles:  Hunter  System  Is  Not 
Appropriate  for  Navy  Fleet  Use,  1  December  1995 

The  GAO  reviewed  a  variant  of  the  $4.2  billion  joint  UAV  program  planned  for 
use  on  Navy  amphibious  assault  ships  to  provide  reconnaissance,  target  acquisition,  and 
other  military  missions. 

Needs  Determination 

The  GAO  finding  was  that,  “The  Joint  Tactical  UAV  Projects  Office  is 
proceeding  with  the  acquisition  of  the  Hunter  shipboard  variant  even  though  all  Navy 
fleet  commanders  have  stated  that  they  do  not  want  the  system  on  Navy  ships.”  [Pg  1] 
The  fleet  commanders  opposed  the  Hunter  UAV  because  of  the  adverse  impact  it  would 
have  had  on  all  other  flight  operations  from  their  ships,  and  because  its  performance 
capability  was  inadequate  for  their  needs. 

Despite  fleet  concerns,  the  Chief  of  Naval  Operations  planned  to  proceed 
with  procurement  of  the  shipboard  variant. 

7.  The  DoD  IG  Report  96-111,  Allegations  Involving  the  Procurement  of 
the  Hunter  Unmanned  Aerial  Vehicle-Short-Range  System,  7  May  1996 

From  1990  to  1996  the  GAO  issued  six  negative  reports  concerning  the  Hunter 
UAV  Short  Range  program  system  deficiencies,  some  of  which  are  documented  above. 
The  first  report  in  September  1990  indicated  that  the  Navy’s  testing  strategy  for  the  UAV 
would  not  be  in  a  realistic  operational  environment,  and  that  full-rate  production  would 
begin  prior  to  the  system  meeting  the  Navy’s  requirements.  The  September  1992  report 
indicates  that  adequate  user  tests  were  not  performed  in  a  realistic  operational 
environment  to  assure  that  critical  system  performance  capabilities  satisfactorily  existed. 
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The  December  1993  report  indicates  that  the  system  didn’t  meet  milestone  criteria  for 
entry  into  TRIP,  and  that  testing  had  not  been  done  to  ensure  that  operational 
requirements  were  met.  The  June  1995  report  states  that  the  system  was  logistically 
unsupportable,  and  that  tests  identified  serious  performance  problems.  The  September 
1995  report  states  that  the  acquisition  demonstrates  the  adverse  consequences  of  initiating 
production  without  first  obtaining  adequate  assurance  of  satisfactory  system  performance. 
This  last  report  of  May  1996  states  that:  “The  audit  partially  or  fully  substantiated  the 
allegations  that  the  TRIP  system  did  not  conform  to  contract  requirements,  operator 
safety  was  at  risk,  reliability  was  inadequate  and  the  system  was  never  subjected  to 
operational  testing.”  [pg.  1] 

In  January  1996,  the  Under  Secretary  of  Defense  (AT&L)  terminated  any  future 
Hunter  system  acquisitions. 

a.  Needs  Determination 

The  Hunter  system  requirements  were  not  met.  As  the  DoD  IG  report 
states,  “The  Government  bought  the  first  of  the  seven  TRIP  systems  knowing  major 
performance  deficiencies  existed  that  made  the  system  clearly  fall  short  of  contractually 
defined  requirements.  The  system  has  numerous  hardware  shortages  that  history  shows 
that  the  contractor  will  never  satisfy.”  [pg.  4]  It  can  be  seen  that  the  PMO  staunchly 
supported  the  program  as  demonstrated  by  the  fact  that  the  Government  contracting 
officer  issued  34  waivers  to  prevent  program  delays  due  to  specification 
nonconformance.  The  Hunter  Project  Office  also  approved  257  engineering  design 
changes  and  125  specification  changes  to  resolve  inconsistencies. 

b.  Reliability 

The  DoD  IG  found  that  the  Hunter  Unmanned  Vehicle  TRIP  did  not  have 
adequate  reliability.  As  the  DoD  IG  report  stated,  “The  Hunter  system  did  not  meet  five 
of  nine  contractual  requirements  for  reliability,  availability,  maintainability,  and  built-in- 
test.”  [pg.  7]  Soldiers  and  military  instructors  using  the  system  also  said  that  it  was 
unreliable.  The  Project  Office  continued  to  support  the  program,  indicating  that  “test 
results  may  not  reflect  the  system’s  long-term  averages  for  the  time  to  repair.”  [pg.  8] 
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c.  Test  and  Evaluation 

The  DoD  IG  found  that  the  Hunter  Unmanned  Vehiele  TRIP  was  not 
subjeeted  to  realistie  operational  testing.  The  Projeet  Offiee  subjeeted  the  prototypes  to 
limited  user  tests,  but  never  performed  any  operational  testing  on  the  TRIP  systems.  The 
DoD  IG  stated, 

The  teehnical  tests  were  done  in  a  eontrolled,  sterile  environment  and  any 
advertised  results  were  not  representative  of  the  system’s  true  eapabilities 
and  real  limitations.  The  system  was  never  subjeeted  to  test  seenarios  that 
replicated  the  true  conditions  the  hardware  would  encounter.  Credible 
operational  tests  have  never  been  accomplished,  [pg.  9] 

8,  DoD  IG  Report  95-193,  DoD  Hotline  Allegations  Regarding  the  Naval 
Special  Warfare  Patrol  Coastal  Ship  and  the  Rigid  Inflatable  Boat  Acquisition 
Programs,  15  May  1995 

At  the  start  of  the  coastal  ship  portion  of  this  program,  the  Chief  of  Naval 
Operations  (CNO)  was  considering  three  alternative  approaches:  a  commercial  item 
approach  using  existing  patrol  craft  technology,  modification  of  an  existing  110-foot 
patrol  craft,  and  new  development.  The  CNO  rejected  new  development  due  to  cost  and 
time  constraints.  The  modification  approach  was  also  rejected,  since  the  110-foot  craft 
did  not  meet  all  the  operational  performance  requirements.  Finally,  the  CNO  approved  a 
Non- Acquisition  Program  Definition  Document,  which  “defines  and  gives  direction  to 
Advanced  and  Engineering  Development  programs  that  explore  technologies  or  integrate 
systems  not  directly  related  to  a  procurement.”  [pg.  7]  The  Non-Acquisition  Program 
Definition  Document  determined  that,  “The  boat’s  performance  and  configuration 
requirements  were  to  be  determined  based  on  existing  craft  in  the  commercial 
marketplace.”  [pg.  7] 

Thus,  it  was  determined  without  further  review  that  a  commercial  item  program 
would  be  selected.  A  firm-fixed  price  contract  was  awarded  for  13  patrol  craft  using  the 
commercial  item  approach.  Over  a  year  later,  the  Supervisor  of  Shipbuilding  imposed 
General  Ships  Specifications  (GSS)  requirements  on  the  patrol  craft  contractor.  This 
caused  the  program  to  evolve  from  commercial  item  to  a  developmental  acquisition 
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program.  The  GSS  requirements  ehange  contributed  to  greater  than  40  percent  cost 
growth  within  the  program. 

Similarly,  the  Rigid  Inflatable  Boat  (RIB)  program  evaluated  four  boats  as 
commercial  item  candidates.  None  of  the  boats  met  operational  requirements.  After  the 
evaluation,  the  PM  for  Support  Ships,  Boats,  and  Crafts  directed  the  Naval  Surface 
Warfare  Center  (NSWC),  “to  develop  a  technical  data  package  for  a  33  foot  (10-meter) 
RIB.”  [pg.  8]  This  directive  effectively  changed  the  program  from  a  commercial  item 
program  to  a  developmental  acquisition  program. 

The  DoD  IG  found  that  the  Navy  deviated  from  the  commercial  item  acquisition 
approach  in  both  cases,  without  conducting  the  necessary  risk  analyses  to  support  the 
decision.  The  Navy  had  modified  the  ship  requirements  beyond  the  point  at  which  a 
commercial  item  could  satisfy  them,  and  then  awarded  a  firm-fixed  price  contract  for  the 
boat  without  adequate  risk  analyses  to  support  the  decision. 

a.  Needs  Determination 

In  the  coastal  ship  program,  the  Navy  combined  requirements,  which  led 
to  the  need  for  a  larger  platform.  There  was  no  detailed  requirements  review  with  the 
program  sponsor  or  operators.  The  increased  requirements  necessitated  changes  beyond 
which  a  commercial  item  acquisition  approach  could  prove  satisfactory. 

In  the  RIB  program,  the  requirements  within  the  Operational 
Requirements  Document  (ORD)  were  more  restrictive  than  those  within  the  Non- 
Acquisition  Program  Definition  Document.  This  should  have  caused  the  Naval  Special 
Warfare  Rigid  Inflatable  Boat  program  office  to  determine  areas  in  which  requirements 
could  have  been  compromised  without  loss  of  mission  effectiveness. 

b.  Design 

Specifications  for  the  RIB  were  based  on  commercial  item  subsystems  and 
components.  However,  their  integration  into  a  functioning  boat  exceeded  the  design  and 
performance  capabilities  of  existing  vessels. 

The  RIB  design  process  was  accelerated  by  the  RIB  PM  from  28  weeks  to 
13  weeks,  but  no  time  was  allowed  to  assess  or  minimize  the  risk  that  this  created  in  the 
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acquisition  process.  It  was  later  determined  by  the  Naval  Sea  Systems  Command 
(NSSC)  that  the  design  should  not  have  been  aeeelerated. 

c.  Risk  Analysis 

The  NSCC  did  not  perform  an  adequate  risk  analysis  that  would  have 
revealed  the  high  technical  risk  caused  by  the  failure  of  eommereial  vessels  to  meet  both 
the  ORD  and  the  greatly  eompressed  aequisition  schedule. 

d.  Test  and  Evaluation 

There  was  no  first  artiele  contraet  for  the  TRIP  of  a  limited  number  of 
boats  prior  to  full-rate  production  for  testing  purposes.  Limiting  the  initial  produetion 
eontract  to  the  first  two  boats  for  test  purposes  would  have  minimized  the  produetion  risk 
for  the  remaining  boats.  Beeause  there  was  no  first  artiele  test  eontraet  for  the  TRIP  of  a 
limited  number  of  boats  prior  to  full  rate  produetion,  there  was  no  operational  testing, 
either.  Eliminating  testing  inereased  the  produetion  risk  that  boats  would  not  meet  user 
operational  requirements,  and  would  then  have  to  be  modified  later  at  greater  expense 
and  time. 

D,  COMMERCIAL  ITEMS  PROBLEMS,  ISSUES  AND  CONSEQUENCES 
SUMMARY 

The  table  below  summarizes  the  problems/issues  from  this  ehapter  as  a  “risk”, 
and  the  “potential  eonsequenee”  of  the  unresolved  risk. 


Table  1  Commercial  Item  Consequence  Matrix 


RISK _ POTENTIAL  CONSEQUENCE 


Needs  Determination 

-  User  requirements  improperly 
identified  or  misunderstood 

-  Requirements  allowed  to  grow;  i.e., 
requirements  are  gold  plated. 


-  Need  unsatisfied;  wrong  product  produced 
and  purehased;  resourees  wasted;  possible 

program  elimination. 

-  “Best  value”  solution  will  not  be  produeed; 

impaets  to  eost,  sehedule,  teehnieal 
performance;  DoD  resourees  wasted. 


41 


Table  1  (continued)  Commercial  Item  Consequence  Matrix 


RISK _ POTENTIAL  CONSEQUENCE 


Acquisition  Stratesv 

-  Incorrect  streamlining  aequisition 
strategy  used. 

-  Items  procured  without  key  acquisition 
phases  eomplete;  impacts  to  program  cost, 
schedule  and  teehnical  performanee;  DoD 
resources  wasted. 

Market  Investigations 

-  Inadequate  or  no  market 
investigation  performed. 

-  Wrong  item  proeured,  “best  value”  item  not 
procured,  impact  to  cost,  schedule  and 
technieal  performance,  DoD  resources  wasted. 

Concent  Formulation 

-  Inadequate  or  no  concept 
formulation  package  prepared. 

-  Best  teehnical  solution  not  selected;  impact 
to  cost,  and  schedule,  DoD  resourees  wasted. 

Risk  Analysis 

-  A  risk  analysis  was  not  performed 

-  Impaet  to  eost,  schedule  and  technical 
performance;  DoD  resourees  wasted;  the 
program  may  be  eliminated. 

Manage  Integration 

-  Inaecurate  assessment  of  program 
integration  risk. 

-  R&D,  or  T&E  not  performed;  impact  to  cost, 
schedule  and  teehnieal  performance;  DoD 
resourees  wasted;  the  program  may  be 
eliminated. 

Test  &  Evaluation 

-  Testing  was  inadequate  or  was  not 
performed. 

-  Best  teehnieal  solution  not  developed; 
performanee,  reliability  &  maintainability 
goals  not  met;  impacts  to  total  LCC; 
operational  effectiveness  &  suitability  not 
proven;  impacts  to  program  cost,  schedule,  and 
technical  performance;  potential  mission 
failure;  DoD  resources  wasted. 
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Table  1  (continued)  C  ommercial  Item  Consequence  Matrix 


RISK 

POTENTIAL  CONSEQUENCE 

Reliability 

-  The  program  reliability  was 
inadequate. 

-  No  quantitative  reliability 
requirements  established. 

-  Contraetor’s  reliability  estimates  not 
verified. 

-  RAM  is  adversely  impacted;  program’s  total 
LCC  is  adversely  impacted;  potential  mission 
failure;  DoD  resources  are  wasted. 

-  Inadequate  reliability  testing  prior  to 
eontract  award. 

Support  Requirements 

-  Ineorreet  program  support  eoneept 
is  seleeted. 

-  Inability  to  support  the  system  during  a 
wartime  environment;  mission  failure; 
program’s  total  LCC  is  adversely  impacted; 
DoD  resources  are  wasted. 

-  Inadequate  logistics  support  is 
planned. 

-  Sub  optimal  logistics  support,  increased 
system  LCC,  DoD  resources  wasted. 

Tech  Data  -  Competitive 
Reprocurement 

-  Data  adequate  for  reprocurement  is 
not  obtained. 

-  A  sole  source  vendor  must  be  used; 
reprocurement  costs  are  increased;  an  inability 
to  obtain  parts;  potential  mission  failure;  DoD 
resources  are  wasted. 

Design 

-  The  design  was  underestimated. 

-  The  best  design  will  not  be  completed; 
impacts  to  program  cost,  schedule  and 
technical  performance. 

-  The  design  process  was  incorrectly 
accelerated. 

-  Impact  to  program  cost,  schedule  and 
technical  performance. 

43 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


44 


IV.  ANALYSIS  -  COMMERCIAL  ITEM  USAGE  EXPERIENCE  & 

PROBLEMS 


A.  INTRODUCTION 

In  this  analysis  chapter,  all  of  the  possible  eauses,  as  well  as  the  underlying 
known  eauses,  of  the  problems  and  issues  in  the  previous  ehapter  will  be  presented  and 
analyzed.  It  is  important  to  determine  the  potential  eritieal  eauses  of  the  problems  and 
issues  in  order  to  arrive  at  a  eorreetive  aetion,  or  “best  praetiee,”  and  thus  to  provide  a 
template  whieh  PMs  ean  use  to  preelude  similar  oeeurrenees  in  the  future. 

The  analysis  will  be  presented  as  an  array,  in  the  same  manner  as  it  was  in  the 
previous  ehapter.  A  statement  of  eaeh  issue  from  the  previous  ehapter  will  be  followed  by 
examples  to  demonstrate  some  of  the  known  eauses  of  that  partieular  issue.  At  the  end  of 
this  ehapter  is  a  table  summarizing  all  the  known  and  potential  eauses,  whieh  have  been 
identified  from  the  reports  that  were  the  basis  of  this  researeh,  as  eited  in  Chapter  III. 

B,  ROOT  CAUSE  PROBLEMS  AND  ISSUES 

For  the  issues  eited  below  in  paragraph  C,  there  are  several  reeurring  eauses  that 
underlie  nearly  every  issue.  These  eauses  ean  be  ealled  “root  eauses,”  beeause  they 
appear  to  be  at  the  terminus,  root,  or  bottom  of  the  explanation  for  many  of  the  issues. 
Several  root  eauses  have  been  identified,  as  follows: 

1.  Inadequate  Education  or  Training  within  the  Acquisition  Community 

This  can  always  be  used  as  an  excuse  for  errors  in  judgment.  Because  their 
leadership  has  told  acquisition  personnel  that  they  must  use  commercial  items,  they 
consider  this  edict  as  justification  for  the  elimination  of  critical  steps  in  the  acquisition 
process  when  acquiring  commercial  items.  Without  the  knowledge  (that  education  or 
training  would  give  them)  of  the  intricacies  or  difficulties  of  each  critical  part  of  the 
acquisition  process,  acquisition  personnel  may  not  understand  that  the  process  cannot  be 
abbreviated  or  eliminated  without  adequate  investigation  and  justification.  Consequently, 
they  suffer  the  risk  of  making  a  disastrously  poor  decision. 
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Because  commercial  item  usage  is  fairly  new,  there  are  many  gray  areas  with 
respect  to  its  correct  usage  in  the  acquisition  process.  Incorrect  usage  happens  frequently 
enough  to  make  training  and  education  an  important  issue. 

2.  Inadequate  Use  of  Lessons  Learned 

In  several  cases  previously  cited,  errors  noted  in  audit  reports  were  repeated.  This 
indicates  a  serious  failure  to  learn  from  previous  mistakes.  The  commercial  item  advocate 
for  the  Army  was  supposed  to  accumulate  lessons  learned,  but  failed  to  do  so.  Within  the 
various  Army  Audit  Agency  reports,  similar  errors  occurred  time  after  time,  indicating 
that  lessons  learned  are  not  being  applied  to  new  Army  acquisition  situations. 

LTC  Michael  Bonheim  indicated  in  his  paper  that  in  the  area  of  market 
investigations,  there  is  no  centralized  database  for  the  accumulation  of  lessons  learned. 

3.  Incorrect  Culture-Driven  Decision  Making 

At  the  root  of  many  poor  decisions  within  the  acquisition  corps  seems  to  be  the 
“culture”  that  drives  the  decision.  That  is,  the  decision  is  a  foregone  conclusion,  given  the 
structure  and  forces  that  impinge  upon  the  PM  during  his  decision-making  process,  as 
previously  noted  in  Chapter  III.  This  area  needs  to  be  researched  in  greater  depth. 

C.  ANALYSIS  OF  PROCESS  PROBLEMS  AND  ISSUES 

Noted  below  are  examples  that  highlight  the  major  causes  of  the  issues  identified 
in  commercial  item  usage. 

1.  Needs  Determination 

Issue:  Needs  or  user  requirements  have  been  improperly  identified, 
misunderstood,  or  ignored. 

The  Navy  coastal  ship  program  started  with  a  requirement  to  use  a  non¬ 
acquisition  approach  to  “base  the  acquisition  on  existing  craft  in  the  commercial 
marketplace.”  [DoD  IG,  pg.  7,  15  May  95]  Because  General  Ships  Specifications 
(military  specifications)  were  imposed  on  the  contractor  one  year  later,  the  subsequent 
acquisition  decision  changed  from  the  original  goal  of  procuring  a  commercial  vessel  to 
that  of  procuring  a  vessel  specifically  designed  to  comply  with  military  specifications. 
This  decision  was  made  without  either  risk  analysis  or  cost  analysis  to  support  the 
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decision.  Certainly,  the  typical  acquisition  process  was  not  followed  in  this  case.  The 
causes  of  this  procedural  omission  could  have  been  any  of  the  following:  inadequate 
acquisition  education  or  training,  cultural  issues  that  stressed  speed,  or  cutting  corners 
and  costs  at  the  expense  of  the  accurate  performance  of  the  acquisition  process.  Another 
potential  cause  of  this  procedural  omission  could  have  been  the  powerful  influence  that 
culture  has  upon  the  way  we  perform  and  operate.  The  PM’s  culture  is  one  that  is  cost, 
schedule,  and  funding  driven,  each  of  these  being  success  oriented  -  sometimes 
constraining  or  preventing  performance  of  the  very  activities  that  would  keep  the  PM  out 
of  difficulty. 

In  the  Navy  UAV  case,  the  Project  Office  blatantly  ignored  the  fleet  commanders’ 
dissatisfaction  with  the  Hunter  UAV.  The  causes  here  could  have  been  many,  such  as  not 
having  a  clearly  written  ORD  that  would  have  identified  the  need  to  minimize 
interference  with  existing  onboard  flight  operations,  inadequate  use  of  IPT’s  in  the 
acquisition  process  that  should  have  included  the  warfighter  as  a  team  member, 
inadequate  training  or  education  of  the  Projects  Office  personnel,  or  a  culture-driven 
decision  by  the  PM  to  ignore  the  warfighter  in  the  interest  of  swift  project  completion. 

In  the  PM-PSE  case,  a  variety  of  users  had  differing  requirements,  but  no 
experience  or  training  in  defining  these  requirements.  The  varied  causes  identified  here 
are  as  follows:  inadequate  teaming  between  user  and  PM,  inadequate  training  and/or 
education  of  the  user  in  the  acquisition  process,  lack  of  standardization  of  the  security 
equipment,  lack  of  a  centralized  force  protection  database,  and  failure  of  the  users  to  be 
flexible  enough  to  consider  modifying  their  requirements  in  order  to  more  easily  utilize 
commercial  force  protection  items. 

Issue:  Requirements  were  allowed  to  change  or  grow,  Le.,  requirements  were 
“gold  plated.  ” 

In  the  coastal  ship  program,  the  Navy’s  requirements  creep  was  due  in  large  part 
to  combining  requirements,  which  led  in  turn  to  the  need  for  a  larger  platform,  causing 
the  length  of  the  vessel  to  grow  from  110’  to  170’  in  length.  This  was  probably  caused  by 
not  having  a  detailed  requirements  review  with  the  program  sponsor  or  operators,  as  was 
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mentioned  in  the  audit.  In  other  words,  there  was  no  IPX,  and  therefore,  no  coordination 
among  the  stakeholders,  as  would  have  occurred  had  there  been  an  IPX.  An  IPX  would 
have  incorporated  the  teaming  tenets  of  cooperation,  coordination,  consensus,  and 
decision  streamlining.  Additional  causal  contributors  were  lack  of  a  clearly  written  ORD, 
lack  of  adequate  acquisition  planning,  and  inadequate  market  research  in  the  beginning 
phase  of  the  program. 

2,  Market  Investigation 

Issue:  There  were  inadequate  or  no  market  investigations  performed. 

At  the  Natick  RD&E  Center,  there  was  a  policy  to  leave  market  investigations 
either  incomplete  or  undocumented,  in  order  to  “streamline”  or  speed  up  the  process. 
Xhis  was  done  without  apparent  concern  for,  or  knowledge  of,  the  negative  impact  that 
inadequate  market  investigations  would  have  on  the  efficiency  and  effectiveness  of  the 
material  selection  process.  Xhis  exemplifies  either  culturally  influenced  decision-making 
in  order  to  incorrectly  “streamline”  the  process  to  save  time  and  money,  or  inadequate 
training  and/or  education  in  preparation  for  the  acquisition  process,  which  would  have 
enabled  those  involved  to  make  the  correct  decisions  from  the  result  of  their  market 
investigations.  Another  alternative  is  that  organizations  might  have  known  the  right  thing 
to  do,  but  were  so  resource  constrained  that  they  couldn’t  get  the  documentation  done. 

Xhe  lack  of  a  database  for  market  investigations  was  documented  by  PM-PSE, 
EXC  Bonheim,  and  was  also  an  issue  in  Dr.  Gansler’s  USD(AX&E)  memorandum  of 
January  5  2001,  which  is  quoted  as  follows: 

Xo  help  overcome  these  barriers  to  accessing  commercial  items,  I  am 
taking  the  following  actions:  ...  Requesting  that  the  IPX  determine  the 
feasibility  of  establishing  a  pilot  program  so  that  the  Services  and 
Agencies  may  collect  market  research  and  Commercial  Item 
Determinations  in  a  central  database,  or  developing  tools  to  assist  in 
ensuring  commercial  item  determinations  are  reasonably  consistent.  I 
request  that  the  recommendation  regarding  this  action  be  presented  to 
DUSD  (AR)  within  90  days  of  the  date  of  this  memorandum.  [DoD. 
USD(AX&E),  5  January  2001] 

Xhis  suggests  an  awareness  within  the  acquisition  community  that  there 
are  “issues”  related  to  ease  of  access  to  both  commercial  item  information  as  well 
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as  usage  data  within  the  aequisition  eommunity,  which,  if  readily  available,  might 
make  the  process  more  efficient.  This  commercial  item  market  database  could 
also  capture  the  valuable  lessons  learned,  which  seem  to  be  lost  to  successive 
generations  of  acquisition  members. 

3,  Concept  Formulation 

Issue:  There  were  inadequate  or  no  concept  formulation  packages  prepared. 

Examples  demonstrating  this  issue  are  to  be  seen  in  the  19  March  1990  Army 
Audit  Agency  report.  This  report  is  about  Natick  RD&E  Center’s  lack  of  preparation  of 
the  concept  formulation  packages  for  two  projects,  which  resulted  in  increasing  costs  and 
schedules  for  both  projects.  One  example  is  the  saltwater  purification  project,  for  which  a 
tradeoff  analysis  would  have  identified  the  Naval  system  as  a  potential  alternative,  thus 
providing  great  time  and  cost  savings  for  the  Army  project.  The  second  example  is  the 
case  of  the  five-soldier  crew  tent  project,  where  a  concept  formulation  package  with  its 
tradeoff  analysis  would  have  eliminated  the  need  to  test  five  tents,  three  times  each. 

4,  Risk  Analysis 

Issue:  A  risk  analysis  was  not  performed. 

The  decision  to  eliminate  risk  analysis  in  the  Navy’s  coastal  ship  program  can  be 
attributed  to  lack  of  training,  lack  of  education,  or  lack  of  attention  to  lessons  learned  in 
the  acquisition  process;  alternatively,  it  might  have  been  culture-driven  in  the  mistaken 
premise  that  time  would  be  saved  if  this  step  in  the  process  were  eliminated.  Although 
there  is  a  Systems  Acquisition  Management  curriculum  (816)  taught  at  the  Naval 
Postgraduate  School,  Naval  officers  are  not  enrolled.  This  is  due  to  the  fact  that  a  military 
specialty  code  for  Systems  Acquisition  Management  does  not  exist  for  Naval  officers. 
Consequently,  the  lack  of  specific  training  or  education  in  the  acquisition  field  is 
understandable  as  a  potential  cause  for  the  elimination  of  risk  analysis  in  their  acquisition 
process. 

5,  Acquisition  Strategy 

Issue:  The  acquisition  process  was  incorrectly  streamlined,  resulting  in  the  use 
of  an  incorrect  acquisition  strategy. 
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In  all  of  the  U.S.  Army  Audit  Agency  reports,  incorrect  acquisition  strategy  was  a 
significant  finding. 

In  the  Natick  Center  report  of  19  March  1990,  the  strategies  were  inconsistent 
with  the  categories  of  commercial  items  being  used.  This  is  an  indication  of  either 
inadequate  training  or  lack  of  education  in  the  acquisition  process,  crucial  for  the 
identification  of  each  acquisition  category  (i.e.,  commercial  item,  modified  commercial 
item,  nondevelopmental  item,  or  developmental  item);  or  the  premeditated  incorrect 
streamlining  of  the  process  to  save  money  and  time  for  culture-driven  reasons. 

In  the  11  September  1990  U.S.  Army  Audit  Agency  report,  key  phases  and 
critical  steps  of  the  acquisition  process  were  eliminated  without  explanation  or 
justification.  This  is  an  indication  of  either  a  lack  of  training  and/or  education  regarding 
the  importance  and  necessity  of  each  part  of  the  acquisition  process,  or  culture-driven 
poor  decisions  in  order  to  save  the  program  time  and/or  money. 

In  the  17  June  1991  report  of  the  U.S.  Army  Audit  Agency,  the  incorrect 
acquisition  strategy  was  used  because,  as  the  five  agencies  audited  in  this  report  told  the 
Army  Audit  Agency,  they  were  directed  to  use  commercial  items,  and  therefore  believed 
this  relieved  them  of  performing  the  necessary  analysis  to  determine  and  support  strategy 
decisions.  This  is  an  indication  of  lack  of  training  and/or  education  concerning  the 
importance  and  necessity  for  each  part  of  the  acquisition  process.  It  is  also  an  example  of 
how  important  “culture”  is  in  decision  making.  The  fact  that  agencies  are  told  to  do 
something  does  not  relieve  them  of  the  responsibility  to  “think,”  and  thus  make  well- 
chosen  decisions  instead  of  poor  ones. 

6,  Design 

Issue:  The  design  was  underestimated. 

The  design  issue,  exemplified  by  the  Coastal  Ship  project,  can  be  attributed  to 
inadequate  training  or  education  on  the  part  of  the  Navy  design  estimation  process  when 
utilizing  commercial  items;  or  it  could  have  been  caused  by  a  culture-driven  need  to 
reduce  the  time  and/or  cost  of  the  acquisition. 

Issue:  The  design  process  was  incorrectly  accelerated. 
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As  an  example  of  this  issue,  the  PM  aceelerated  the  Rigid  Inflatable  Boat  (RIB) 
design  proeess  from  28  weeks  to  13  weeks  without  any  analysis  given  to  the  risk  or 
eonsequenees  of  doing  so.  The  elear  goal  of  the  PM  was  to  reduee  the  time  required  of 
the  project’s  design  task.  This  issue  could  be  attributed  to  inadequate  training  or 
education  on  the  part  of  the  Navy  concerning  the  use  of  risk  analysis  in  the  acquisition 
process;  or  it  could  have  been  caused  by  the  culture-driven  need  of  the  PM  to  reduce  the 
schedule  and/or  cost  of  the  program. 

7.  Management  of  Integration 

Issue:  Integration  projects  associated  with  high  risk  for  a  program  were  not 
managed  properly. 

An  example  of  this  issue  was  published  in  the  U.S.  Army  Audit  Agency’s  17  June 
1991  report,  where  it  was  noted  that  there  were  four  complex  commercial  item 
integration  projects  for  a  total  cost  of  $1 .7  billion,  none  of  which  had  a  testing  phase  prior 
to  awarding  their  production  contract.  The  commercial  items  were  all  off-the-shelf 
components.  However,  the  integrated  systems  were  not  off-the-shelf  and  were  highly 
complex  as  well.  For  these  four  projects,  there  was  little  or  no  information  to  prove  that 
the  integrated  systems  would  successfully  work  together  to  meet  the  Army’s 
requirements.  The  failure  to  manage  the  integration  of  commercial  items  within  a 
program  can  be  attributed  to  either  lack  of  training  and/or  education,  or  a  culture-driven 
desire  to  reduce  the  time  and/or  cost  of  the  program. 

8,  Test  and  Evaluation 

Issue:  There  was  inadequate  testing  or  no  testing  performed. 

An  example  of  this  issue  is  the  U.S.  Army  Audit  Agency  11  September  1990 
report  on  the  $643  million  Regency  Net  project,  which  involved  a  mix  of  commercial  as 
well  as  previously  developed  military  items,  all  of  which  had  to  be  integrated  via 
complex  software  development.  This  project  included  both  a  basic  contract  as  well  as 
four  production  options.  It  was  awarded  at  a  cost  of  $300  million  before  any  prototypes 
were  tested.  Another  project  from  the  Army  Audit  Agency  11  September  1990  report  on 
Mobile  Subscriber  Equipment  again  involved  integration  of  commercial  items  with 
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previously  developed  military  items,  as  well  as  the  development  of  eomplex  software.  It 
was  not  tested  until  $1.3  billion  of  the  basie  production  contract  had  been  obligated, 
along  with  two  production  options.  The  rationale  given  to  the  Army  Audit  Agency  was  as 
follows:  “Acquisition  strategies  for  the  projects  stated  that  extensive  testing  was  not 
needed  because  a  nondevelopmental  item  approach  was  used  or  that  a  contractor 
demonstrated  the  item  before  the  production  contracts  were  awarded.”  [U.S.  Army  Audit 
Agency,  pg  21,  11  September  1990]  However,  the  contractor  demonstrations  did  not 
provide  sufficient  evidence  that  the  items  had  met  the  Army  operational  requirements. 
The  failure  to  properly  test  and  evaluate  commercial  items  within  a  program  can  be 
attributed  to  lack  of  training  or  education,  or  can  be  caused  by  a  culture-driven  need  on 
the  part  of  the  PM  to  reduce  the  time  and/or  cost  of  the  program  by  eliminating  or 
reducing  testing. 

9.  Logistics  Support 

a.  Reliability 

Issue:  The  program  reliability  was  inadequate. 

An  example  of  this  issue  is  the  PM-PSE  paper,  which  indicated  that 
vendor  claims  proved  inaccurate.  This  can  be  traced  to  either  inadequate  early  testing  of 
the  products  or  inadequate  training  and/or  education  concerning  the  need  for  adequate 
evaluation  and  testing  of  a  commercial  item  in  order  to  confirm  the  vendor’s  reliability 
claims. 

Another  example  of  inadequate  program  reliability  is  the  DoD  IG  report  of 
7  May  1996,  which  showed  that  the  Hunter  UAV  program  continued  to  receive  PM 
support  even  though  it  did  not  have  adequate  reliability.  This  can  be  attributed  to  the 
PM’s  culture-driven  syndrome  to  protect  his  program  at  all  costs,  in  this  case  by  avoiding 
early  testing  that  might  have  demonstrated  reliability  flaws,  resulting  in  a  program 
funding  cut  and,  ultimately,  delays  or  even  termination  of  the  program. 

The  primary  cause  of  inadequate  reliability  can  be  traced  to  inadequate 
testing  early  in  the  program.  A  major  reason  for  inadequate  testing  is  often  the  cultural 
aversion  on  the  part  of  the  PM  to  both  testing  and  testers,  as  previously  mentioned  in 
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Chapter  III,  section  B-7.  Another  potential  cause,  as  noted  above,  is  inadequate  training 
and/or  education  regarding  the  importance  of  attaining  adequate  system  reliability. 

Issue:  No  quantitative  reliability  requirements  established. 

This  issue  is  exemplified  by  the  U.S.  Army  Audit  Agency  reports  of  11 
September  1990  and  17  June  1991,  which  both  state  that  many  projects  had  no 
quantitative  reliability  requirements  established.  In  the  absence  of  reliability 
requirements,  there  is  no  basis  upon  which  to  perform  confirmatory  tests.  However,  in 
some  cases,  with  or  without  a  reliability  requirement,  contractor  reliability  estimates  were 
verified,  but  not  early  enough  to  allow  time  for  corrective  action.  In  other  cases, 
reliability  testing  was  not  conducted  at  all. 

The  cause  for  not  establishing  reliability  requirements  could  be  one  or 
more  of  the  following:  PMs  having  been  culture-driven  to  save  the  program  time  and/or 
money;  lack  of  training/education  which  enabled  the  mistaken  belief  that  management’s 
edict  to  use  commercial  items  voids  the  need  for  establishing  reliability  of  commercial 
items;  or  the  lack  of  training  and/or  education  of  users  and  PMs  regarding  the  importance 
of  establishing  reliability  requirements,  especially  for  commercial  items. 

Issue:  Contractor’s  reliability  estimates  not  verified. 

This  issue  is  displayed  in  the  U.S.  Army  Audit  Agency  reports  of  11 
September  1990  and  17  June  1991.  In  both  reports,  vendors’  reliability  estimates  were 
not  verified  in  the  operating  environment.  In  several  cases,  reliability  estimates  had  been 
received,  but  without  the  performance  data  necessary  in  order  to  evaluate  the  contractor 
claims.  In  four  programs,  the  contractor’s  predicted  values  for  the  quantitative 
requirements  were  used,  but  not  verified.  These  contractor-predicted  values  were 
basically  technical  estimates;  they  were  not  based  on  any  real  data  or  analyses  and 
certainly  did  not  reflect  the  operating  environment  and  conditions  where  military  systems 
must  perform. 

The  reasons  for  not  verifying  a  contractor’s  reliability  estimates  are 
attributable  to  one  or  all  of  the  following:  lack  of  training  and/or  education  concerning 
the  importance  of  commercial  item  reliability  verification  through  testing;  culture-driven 
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decision-making  to  save  the  program  time  and  money  by  eliminating  or  reducing  testing; 
or  lack  of  training/education  enabling  the  mistaken  belief  that  management’s  edict  to  use 
commercial  items  voids  the  need  for  verifying  the  reliability  of  commercial  items. 

Issue:  Inadequate  reliability  testing  prior  to  contract  award. 

This  issue  is  exemplified  by  the  U.S.  Army  Audit  Agency  17  June  1991 
report,  as  well  as  both  the  DoD  IG  reports  of  15  May  1995  and  7  May  1996.  They  all 
stated  that  many  projects  had  no  or  inadequate  reliability  testing  prior  to  contract  award. 
In  the  case  of  the  Hunter  UAV,  inadequate  reliability,  among  other  issues,  killed  the 
program. 

Inadequate  reliability  testing  can  be  attributed  to  a  lack  of  training  and/or 
education  on  the  importance  of  commercial  item  reliability  testing  early  in  the  program;  a 
culture-driven  decision  to  save  the  program  time/money;  or  the  lack  of  training  and/or 
education,  enabling  the  mistaken  belief  that  management’s  edict  to  use  commercial  items 
voids  the  need  for  verifying  the  reliability  of  commercial  items  through  testing. 

b.  Support  Requirements 

Issue:  Incorrect/Inadequate  program  support  concept  is  selected. 

The  U.S.  Army  Audit  Agency  report  of  11  September  1990  indicated  that 
for  the  sake  of  expediency,  the  end-item  contractor  was  to  have  provided  both 
maintenance  and  supply  support  for  the  expected  life-cycle  for  seven  of  the  ten  projects 
reviewed.  The  reason  for  lack  of  adequate  logistics  support  in  this  case  was  most  likely 
culture-driven  to  save  the  program  time  and/or  money.  Lack  of  training  and/or  education, 
related  to  the  importance  of  establishing  and  justifying  support  concepts,  also  could  have 
played  a  role. 

c.  Technical  Data  for  Competitive  Reprocurement 

The  U.S.  Army  Audit  Agency  report  of  17  June  1991  indicated  that  14 
projects  reviewed  did  not  acquire  technical  data  suitable  for  reprocurement  of  spare  parts 
or  systems.  The  causes  for  not  obtaining  technical  data  suitable  for  competitive 
reprocurement  of  parts  are  attributed  to:  lack  of  training  and/or  education  regarding  the 
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importance  and  difficulty  of  obtaining  commercial  item  teehnical  data;  or  eulture-driven 
decisions  to  save  the  program  money  during  the  aequisition  phase. 

D,  COMMERCIAL  ITEM  -  CAUSE  -  SUMMARY 

In  the  table  below,  the  problems/issues  are  listed  as  risks,  along  with  the  potential 
eonsequence  of  the  unresolved  risk,  and  their  possible  causes,  as  identified  in  this 
ehapter. 

Table  2  Commercial  Item  Cause  Matrix 


RISK _ POTENTIAL  CONSEQUENCE _ POSSIBLE  CAUSE 


Needs  Determination 

-  User  requirements 
are  improperly 
identified, 
misunderstood  or 
ignored. 

-  Requirements 
allowed  to  grow,  i.e., 
requirements  are  gold 
plated. 

-  Need  unsatisfied;  wrong 
product  produced  and 

purchased;  DoD  resources 
wasted;  possible  program 
elimination. 

-  “Best  value”  solution  will 
not  be  produced;  impacts  to 

cost,  schedule,  technical 
performance;  DoD  resourees 
wasted. 

-  No  acquisition  planning/risk 
analysis  performed;  inadequate 
training/education;  unelear 
ORD;  IPTs  not  used;  culture- 
driven  deeisions;  no  product 
standardization;  no  centralized 
database;  inflexible  reqmts. 
-No  aequisition  planning/IPT; 
needs  allowed  to  grow  too 
easily;  inadequate  ORD; 
insufficient  market  research. 

Acquisition  Strategy 

-  Ineorreet 
streamlining 
aequisition  strategy 
used. 

-  Items  procured  without  key 
aequisition  phases  complete; 
impacts  to  program  cost, 
schedule  and  technical 
performance;  DoD  resources 
wasted. 

-  Lessons  learned  not  utilized; 
no  evaluation  of  streamlining 
deeisions;  inadequate  training/ 
edueation;  incorrect  culture- 
driven  deeisions. 

Market 

Investigations 

-  Inadequate  or  no 
market  investigation 
performed. 

-  Wrong  item  procured,  “best 
value”  item  not  proeured; 
impaet  to  eost,  sehedule  and 
technical  performance;  DoD 
resourees  wasted. 

-  No  early,  robust,  market 
investigation  performed; 
inadequate  training/edueation; 
no  centralized  database/lessons 
learned;  inadequate  funding. 
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Table  2  (continued) 


Commercial  Item  Cause  Matrix 


RISK 

POTENTIAL  CONSEQUENCE 

POSSIBLE  CAUSE 

Concept  Formulation 

-  Inadequate  or  no  concept 
formulation  package 
prepared. 

-  Best  technical  solution 
not  selected;  impact  to  cost, 
schedule  and  technical 
performance;  DoD 
resources  wasted. 

-  Inadequate  training/ 
education/  or  lessons 
learned;  incorrect  culture- 
driven  decisions. 

Risk  Analysis 

-  A  risk  analysis  was  not 
performed. 

-  Impact  to  cost,  schedule 
and  technical  performance; 
DoD  resources  wasted;  the 
program  may  be  eliminated. 

-  No  early,  robust,  market 
investigation  performed; 
inadequate  training  / 
education  /  or  use  of  lessons 
learned;  widespread 
misunderstanding  of  the 
complexity  of  integration. 

Manage  Integration 

-  Inaccurate  assessment  of 
program  integration  risk. 

-  R&D,  or  T&E  not 
performed;  impact  to  cost, 
schedule  and  technical 
performance;  DoD 
resources  wasted;  the 
program  may  be  eliminated. 

-  No  early,  robust,  market 
investigation  performed, 
inadequate  training  / 
education  /  or  use  of  lessons 
learned;  widespread 
misunderstanding  of  the 
complexity  of  integration. 

Test  &  Evaluation 

-  Testing  was  inadequate  or 
was  not  performed. 

-  Best  technical  solution  not 
developed;  performance, 
reliability  &  maintainability 
goals  not  met;  impact  total 
LCC;  operational  suitability 
&  effectiveness  not  proven; 
impacts  to  program  cost, 
schedule,  and  technical 
performance;  potential 
mission  failure  /  program 
elimination;  DoD  resources 
wasted. 

-  Culture-driven  poor 
decisions;  inadequate 
training  /  education  /  or  use 
of  lessons  learned;  lack  of 
funding. 

Design 

-  The  design  was 
underestimated. 

-  The  design  process  was 
incorrectly  accelerated. 

-  The  best  design  will  not 
be  completed;  impacts  to 

program  cost,  schedule  and 
technical  performance. 

-  Impact  to  program  cost, 
schedule  and  technical 

performance. 

-  Inadequate  training  / 
education  /  or  use  of  lessons 
learned;  culture-driven  poor 
decision. 

-  Culture-driven;  inadequate 
training/education/use  of 
lessons  learned. 
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Table  2  (continued) 


Commercial  Item  Cause  Matrix 


RISK _  POTENTIAL  CONSEQUENCE _ POSSIBLE  CAUSE 


Reliability 

-  The  program  reliability 

was  inadequate. 

-  No  quantitative  reliability 
requirements  established. 

-  Contraetor’s  reliability 
estimates  not  verified. 

-  Inadequate  reliability 
testing  prior  to  eontraet 

award. 

-  RAM  is  adversely 
impaeted;  program’s  total 
LCC  is  adversely  impaeted; 
potential  mission  failure; 
DoD  resourees  are  wasted. 

-  Poor  eulture-driven 
deeision;  inadequate 
training  /  edueation  /  use  of 
lessons  learned;  laek  of 
funding. 

Support  Requirements 

-  Ineorreet  program  support 
eoneept  is  seleeted. 

-  Inadequate  logisties 
support  is  planned. 

-  Inability  to  support  the 
system  during  a  wartime 
environment;  mission 
failure;  program’s  total 
LCC  is  adversely  impaeted; 
DoD  resourees  are  wasted. 

-  Sub  optimal  logisties 
support,  inereased  system 
LCC,  DoD  resourees 
wasted. 

-  Poor  eulture-driven 
deeisions;  inadequate 
training  /  edueation  /  use  of 
lessons  learned; 
misunderstanding  of  the 
sehedule  needed  for 
logisties  even  for  CIs/NDIs. 

-  Insuffieient  time  allotted 
for  logisties  support 
planning  /  preparation; 
EMD  phase  eliminated. 

Tech  Data  -  Competitive 
Reprocurement 

-  Data  adequate  for 
reproeurement  is  not 
obtained. 

-  A  sole  souree  vendor  must 
be  used;  reproeurement 
eosts  inereased;  an  inability 
to  obtain  parts;  potential 
mission  failure;  DoD 
resourees  are  wasted. 

-  Poor  eulture-driven 
deeisions;  inadequate 
training  /  edueation  /  use  of 
lessons  learned. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A,  INTRODUCTION 

The  objective  of  this  thesis  has  been  to  study  the  problems  and  issues  experienced 
by  PMs  and  industries  utilizing  commercial  items  in  their  programs.  To  explore  this 
subject,  the  researcher  has  reviewed  and  analyzed  the  history  of  problems  and  issues 
experienced  by  acquisition  programs  using  commercial  items  in  their  programs.  This 
chapter  presents  the  conclusions  of  this  thesis,  offering  recommended  best  practices  for 
utilizing  commercial  items  on  a  program,  as  well  as  suggesting  areas  for  further  research. 

B,  METHODS  AND  PROCESSES  “BEST  PRACTICE”  CONCLUSIONS 

It  appears  that  many  mistakes  or  decisions  have  been  made  due  to  the  fact  that  the 
decision-makers  thought  they  understood  the  process  leading  them  to  make  the  correct 
streamlining  decision.  That  is,  they  thought  that  their  expertise  would  enable  them  to 
bypass  certain  requirements,  or  to  ‘streamline’  the  process  for  the  purpose  of  saving  time 
and  money,  but  instead  costed  their  program  both  time  and  money  due  to  the  resultant 
errors. 

1,  Lessons  Learned 

It  is  imperative  to  learn  from  our  mistakes.  There  are  many  lessons  to  be  learned 
from  the  various  governmental  audits  and  reports,  and  from  the  use  of  commercial  items 
in  each  of  the  services.  These  lessons  learned  should  be  accumulated  in  a  centralized 
database  available  to  the  military  service  PM  community  as  a  web  site,  and  should  also 
be  taught  in  various  acquisition  courses.  This  database  website  should  incorporate  market 
investigation,  research  material,  lessons  learned,  and  the  “best  practices”  to  use  in 
particular  situations  and  circumstances.  It  should  be  not  only  an  information  database 
website,  but  also  a  website  for  learning  about  the  unique  aspects  of  using  commercial 
items  in  the  acquisition  process 

2,  Culture-Driven  Mistakes 

As  noted  previously  in  Chapter  III,  section  seven,  PMs  have  occasionally  been 
driven  by  culture  to  make  negative  and  flawed  decisions  that  have  adversely  impacted 
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their  programs,  and  occasionally  their  careers.  In  order  to  change  these  negative  results, 
it  is  necessary  to  change  the  basic  culture  and  reward  system  that  impinges  upon  the 
PMs’  world. 

The  PM-test  community  relationship  is  a  vivid  example  of  the  adverse  impacts  of 
the  wrong  culture  and  reward  system  at  work.  Rather  than  Congress  killing  a  program 
because  it  has  failed  testing,  the  discovery  of  technical  problems  during  early  testing 
instead  of  during  production  should  be  rewarded,  before  the  cost  to  fix  such  technical 
problems  becomes  increasingly  expensive  and  time  consuming. 

To  foster  a  true  and  lasting  attitudinal  change  of  the  PMs’  aversion  to  T&E  within 
the  acquisition  realm,  it  will  be  necessary  to  change  the  current  culture  and  reward 
system.  The  acquisition  of  knowledge  through  testing,  and  the  subsequent  increase  of  a 
product’s  technological  maturity,  must  be  viewed  by  everyone  in  the  acquisition 
community  as  a  good  and  desirable  result,  and  rewarded  accordingly.  For  example,  DT 
programs  that  first  mature  their  technologies  through  adequate  testing,  thus  allowing  their 
programs  to  enter  the  production  phase  with  subsequent  low-risk  technologies,  should  be 
rewarded  by  full  funding  and  continuation  of  their  programs.  On  the  other  hand, 
programs  that  attempt  to  enter  production  with  high  to  medium-risk  technologies  should 
be  penalized. 

In  many  cases  the  PM/PMO  is  well  aware  of  and  sensitive  to  the  issues  faced  but 
the  institutional  biases  are  overwhelming  and  cause  them  to  make  poor  decisions. 

3.  Training  and  Education 

Training  and  education  is  extremely  important  within  the  acquisition  community. 
The  military  makes  the  higher  education  and  leadership  training  of  their  personnel  a 
career-long  endeavor.  However,  within  the  DoD  civilian  community  that  is  not 
necessarily  the  case,  and  this  must  change.  The  acquisition  of  major  weapon  systems  is 
the  most  complex  environment  within  DoD,  and  all  of  its  personnel  must  be  adequately 
trained  and  educated  continuously  in  order  to  master  its  intricacies,  maintain  current 
knowledge  and  execute  the  acquisition  process  properly. 
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There  appear  to  be  large  gaps  within  the  education  process,  especially  concerning 
the  use  of  commercial  items  and  nondevelopmental  items  within  DoD  acquisition 
programs.  Policy  drives  the  acquisition  process  and  is  far  ahead  of  the  education  of  its 
disciples,  especially  for  commercial  item  usage.  There  is  inadequate  training  for  user 
personnel  to  learn  how  to  write  performance-based  requirements.  There  is  no  risk 
management  training  oriented  toward  commercial  item/nondevelopmental  item  usage. 
There  has  been  inadequate  market  investigation  training.  The  acquisition  training  and 
education  process  should  be  critically  examined  for  such  voids  and  the  deficiencies 
corrected. 

Training  and  education  alone  will  not  correct  the  DoD  commercial  item  usage 
problems.  It  is  imperative  that  the  “culture”  within  DoD  that  drives  PMs  to  make 
negative,  self-defeating  decisions  be  addressed  and  changed.  The  culture  and  reward 
system  must  promote  and  reward  good  decisions.  A  research  article  by  a  professor  at  the 
Naval  Postgraduate  School  about  the  effect  of  training  on  changing  the  culture  within 
DoD  inventory  management  noted,  “To  be  effective,  it  must  be  coupled  with  changes  in 
other  organizational  design  factors  such  as  reward  systems  and  management  control 
systems.  Training  is  only  one  of  many  inputs  to  organizational  behavior.  Trying  to  alter  a 
culture  by  changing  only  one  of  these  factors  is  usually  a  wasted  effort.”  [Kang,  pg.  21] 

4.  Needs  Determination 

Users  initiate  an  acquisition  as  they  conduct  the  requirements  generation  process. 
User-led  multifunctional  teams  need  to  be  formed  as  early  as  possible  to  assist  the  user  in 
fully  grasping  the  nature,  potential  limitations,  and  advantages  of  available  commercial 
items.  Only  through  an  adequate  understanding  of  the  advantages  of  commercial  items 
can  the  users  and  acquisition  community  embrace  the  flexibility  necessary  in  defining 
system  requirements  that  will  encourage  greater  consideration  of  commercial  item  usage. 

5,  Acquisition  Strategy 

It  is  critical  that  both  the  buying  organizations  and  the  acquisition  community 
understand  the  importance  of  devising  farsighted  acquisition  strategies,  ones  that 
adequately  address  the  program  uncertainties  that  can  result  from  the  use  of  commercial 

items.  Acquisition  strategies  should  be  devised  to  target  those  commercial  item 
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uncertainties  with  the  potential  to  impaet  produet  performanee,  quality,  and  logistieal 
support. 

6,  Market  Investigation 

When  market  investigations  are  not  eondueted  properly,  it  is  possible  that  some 
potential  alternatives  will  not  be  identified  and  investigated.  Consequently,  the  best-value 
alternative  might  not  be  selected.  If  the  best-value  is  not  seleeted,  the  result  will  lead  to 
greater  program  eost  and/or  time,  the  selection  of  substandard  technology,  or  the  least 
supportable  material  solution. 

A  good  market  investigation  also  would  identify  deficieneies  in  the  eommereial 
item  that  would  require  modifieation,  and  would  determine  the  extent  of  that 
modification.  Accurate  determination  of  the  neeessary  modifications  is  important;  having 
to  modify  the  seleetion  more  then  antieipated  would  add  both  eost  and  time  to  the 
program.  Extensive  modifieation  of  eommereial  items  within  a  program  would  take  the 
produet  out  of  the  eategory  of  a  eommereial  item,  thus  inereasing  the  program’s  risk. 

Market  investigation,  along  with  eoncept  formulation,  is  a  key  determiner  of  the 
aequisition  strategy.  If  the  market  investigation  is  not  performed  properly,  the  seleeted 
aequisition  strategy  will  probably  not  be  the  most  effeetive  and  effieient  one  to  use. 

Key  personnel  within  the  buying  organization  must  be  adequately  trained  and 
experieneed  in  the  market  investigation  proeess.  In  the  Natick  Laboratory  case  of  the 
snow  and  iee-traversing  equipment,  a  proper  market  investigation  would  have  headed  off 
the  problems  with  the  seleeted  off-the-shelf  items  prior  to  aequiring  and  testing  them.  In 
the  ease  of  the  field  paek,  a  market  investigation  would  have  identified  the  defieieneies 
with  the  eommereial  paek,  whieh  subsequently  required  modification  to  such  an  extent 
that  it  bore  no  resemblanee  to  the  original  eommereial  paek. 

7.  Concept  Formulation 

Without  an  adequate  eoneept  formulation  paekage,  whieh  ineludes  both  a  tradeoff 
determination  and  the  best  teehnieal  approaeh,  it  is  unlikely  that  the  best  alternative  will 
be  seleeted.  This  will  eost  the  program  both  time  and  money,  and  possibly  forego  the  best 
teehnology  as  well.  Sinee  eoneept  formulation  is  one  of  the  determiners  of  the  aequisition 
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strategy,  incorrect  concept  formulation  is  likely  to  lead  to  a  substandard  acquisition 
strategy.  As  the  Army  Audit  Agency  states:  “The  concept  formulation  process  should  be 
conducted  and  documented  for  nondevelopmental  items.”  [U.S.  Army  Audit  Agency,  pg. 
23,  11  Sept  1990] 

Even  in  cases  where  management  directs  the  use  of  commercial  items,  there  are 
always  tradeoffs  and  options  that  need  to  be  identified,  analyzed  and  evaluated  to 
establish  the  best  strategy  for  implementation. 

8,  Risk  Analysis 

Program  risk  will  be  different  when  using  commercial  items  than  in  a  traditional 
developmental  acquisition  program.  Risk  is  fairly  well  defined  for  commercial  items  in 
the  traditional  areas  of  technical  performance,  cost  and  scheduling.  For  example,  the 
commercial  item  market  investigation  will  determine  cost  and  technical  performance,  and 
delivery  dates  are  subject  to  negotiation  and  discussion.  In  the  areas  of  integration, 
operational  suitability  and  supportability,  however,  risk  analysis  should  be  strongly 
emphasized,  not  eliminated. 

The  risk  that  occurs  during  the  integration  of  commercial  parts  must  also  be 
recognized,  and  program  strategy  to  reduce  or  eliminate  that  risk  should  be  established. 

Commercial  items  that  are  to  be  used  in  a  military  environment  significantly 
different  from  that  for  which  they  are  designed  carry  high  risk.  These  items  must  be 
qualified  in  the  operational  environment  in  which  they  will  be  used. 

There  is  inherent  supportability  risk  within  the  commercial  realm  due  to  the  rapid 
change  within  the  commercial  marketplace,  as  previously  discussed.  This  risk  needs  to  be 
recognized  and  planned  for  in  order  to  produce  adequate  total  life-cycle  support  for  the 
commercial  item. 

9.  Design 

The  area  of  design  cannot  arbitrarily  be  rushed  in  order  to  speed  up  the  program. 
Likewise,  the  use  of  commercial  items  does  not  necessarily  mean  that  the  design  process 
has  been  reduced  or  streamlined.  In  fact,  the  incorporation  of  commercial  items  into  a 
design  may  be  more  difficult,  because  all  of  the  design  parameters  of  the  commercial 
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item  may  not  be  known.  This  is  an  area  that  demands  eareful  analysis  before  attempting 
to  eompress  design  sehedules. 

10,  Management  of  Integration 

There  is  widespread  misunderstanding  of  the  eomplexity  of  eommereial  item  and 
nondevelopmental  item  integration  into  a  program.  Integration  projects  cannot  be  treated 
as  normal  low-risk  commercial  item  acquisitions.  Such  integration  efforts  may  not  only 
require  research  and  development,  but  also  may  need  both  a  demonstration  and  a  full- 
scale  development  phase  before  production,  unless  the  integration  risk  is  determined  to 
be  sufficiently  low. 

11.  Test  and  Evaluation 

Test  and  Evaluation  (T&E)  must  be  included  in  the  evaluation  of  both  commercial 
items  and  nondevelopmental  items  to  ensure  that  operational  suitability,  reliability, 
availability  and  maintainability  requirements  are  met.  Only  for  commercial  off-the-shelf 
items  that  are  very  low  risk  should  developmental  T&E  be  allowed  to  be  eliminated  and 
operational  testing  should  always  confirm  the  suitability  of  a  commercial  item  in  the 
military  environment.  As  the  Army  Audit  Agency  indicates,  “More  effective  up-front 
planning  of  independent  test  and  evaluation  is  needed  to  ensure  that  enough  data  is 
obtained  to  fully  evaluate  the  capabilities  of  new  nondevelopmental  items.”  [El.S.  Army 
Audit  Agency,  pg  28,  11  Sept.  1990] 

Testing  should  be  performed  on  commercial  items  before  and  during  production, 
unless  a  definitive  decision  verified  by  contractor  data  or  previous  tests  adequately 
demonstrates  the  item’s  acceptability.  Sufficient  data  must  be  available  to  assess  the 
critical  testing  issues  of  manpower  and  personnel  integration,  reliability,  safety  and 
supportability.  Any  decision  to  reduce  or  eliminate  developmental  testing  should  be 
based  on  independent  evaluation  reports  that  assess  either  the  previous  testing  or  other 
data  to  indicate  how  much  testing  will  be  required.  A  GAO  study  found  that  a  DoD 
weapon  system  frequently  “experiences  Tate  cycle  churn’  or  the  scramble  to  fix  a 
significant  problem  discovered  late  in  development.”  [GAO,  pg  5,  31  July  2000]  This 
Tate  cycle  chum’  is  attributable  to  the  inadequate  exposure  of  technical  problems  early  in 

the  program  via  a  robust  test  and  evaluation  process. 
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Operational  testing  of  a  eommercial  item  in  the  military  environment  it  will  be 
used  in  order  to  verify  operational  suitability,  effectiveness  and  performance  should  never 
be  waived  unless  it  will  be  used  in  the  exact  same  environment  for  which  it  was  designed. 

12,  Logistics  Support 

A  significant  problem  concerning  logistics  support  for  commercial  items  has 
become  apparent.  Products  are  constantly  being  changed  within  the  commercial  item 
industry  to  keep  contractors  competitive.  This  constant  change  drastically  affects  the 
manufacturer  support  for  older  products.  The  manufacturer  may  no  longer  support  older 
obsolete  products,  thus  making  replacement  parts  difficult,  if  not  impossible,  to  obtain. 
As  an  Armed  Forces  Journal  International  magazine  article  puts  it,  “But  today,  the  true 
cost  of  COTS  is  beginning  to  sink  in.  Sure,  DoD’s  R&D  costs  come  down,  but  life-cycle 
costs  under  COTS  need  to  substantially  increase.  Staying  in  step  with  the  commercial 
world’s  technology  churn  will  require  continuous  mini-development  efforts  throughout  a 
program’s  life.”  [Baker,  pg4] 

The  process  of  streamlining  for  commercial  items  or  nondevelopmental  items  can 
result  in  the  elimination  of  the  Engineering  &  Manufacturing  Development  (EMD)  or 
System  Development  and  Demonstration  (SDD)  phase,  which  would  normally  include 
detailed  logistics  planning  and  acquisition  of  the  necessary  logistics  support 
infrastructure.  This  may  result  in  not  allowing  enough  time  for  the  logistics  planning 
process  in  order  to  adequately  develop  and  acquire  the  support  materiel  package, 
maintenance  structure,  organizational  and  support  training  packages,  and  writing  of 
technical  manuals. 

a.  Reliability 

If  a  commercial  item’s  reliability,  as  stated  by  the  vendor,  is  not  verified 
as  meeting  the  user’s  requirements,  then  the  system’s  ECC  may  be  severely  impacted. 
Eower  reliability  greatly  impacts  the  support  costs,  system  availability,  and  thus  the 
mission  accomplishment.  RAM  requirements  must  be  established  early  in  order  to  insure 
adequate  testing  and  verification  of  the  reliability  of  commercial  or  nondevelopmental 
items. 
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Since  a  commercial  or  nondevelopmental  item  has  already  been  designed 
and  developed,  and  its  reliability  already  established,  the  reliability  verification  should  be 
an  operational  assessment  of  the  product  within  the  military  wartime  environment  in 
which  it  will  be  used. 

b.  Support  Requirements 

Because  of  the  volatility  of  commercial  items  with  respect  to  design 
changes,  as  previously  mentioned,  the  availability  of  commercial  item  parts,  components, 
or  systems  may  cease  before  the  intended  military  life-cycle  of  the  commercial  item 
expires.  Therefore,  support  requirements  must  focus  on  the  long-term  maintenance  and 
support  of  commercial  items  when  fielded,  because  Government  access  to  commercial 
item  technical  data  will  most  likely  be  limited. 

c.  Technical  Data  for  Competitive  Reprocurement 

Plans  for  access  to  data  suitable  for  competitive  reprocurement  of 
commercial  items  are  critical.  Obsolescence  within  the  commercial  item  marketplace  is 
usually  more  rapid  and  unpredictable  than  it  is  for  products  developed  specifically  for  the 
Government.  The  commercial  item  marketplace  is  more  volatile;  products  can  be  dropped 
abruptly,  or  changed  and  businesses  can  go  bankrupt  or  be  bought  out  by  other 
companies.  Without  adequate  planning  for  reprocurement,  it  is  assured  that  there  will  be 
trouble  within  the  program  -  and  sooner  then  anyone  might  think. 

C.  COMMERCIAL  ITEM  SUMMARY 

The  inclusion  of  commercial  items  in  the  acquisition  process  is  recognized  as  an 
opportunity  to  save  both  time  and  money.  But  it  is  not  the  Holy  Grail.  It  can  do  all  that 
everyone  expects  it  to  do,  and  may  be  an  excellent  solution  in  many  cases,  but  its  use 
should  be  as  the  result  of  carefiil  analysis,  reasoning  and  multi-functional  study.  As 
Carney  and  Oberndorf  indicate  in  their  research  paper  on  the  use  of  COTS  to  purchase 
software,  “The  critical  point  is  that  using  COTS  components  in  any  given  circumstance 
might  help,  but  is  not  guaranteed  to,  and  such  use  may  even  cause  greater  problems.” 
[pg.  1]  They  propose  what  they  call  the  “ten  commandments,”  which  must  be  considered 
when  using  COTS.  All  of  these  “commandments”  apply  directly  to  the  use  of  commercial 
items  other  than  software. 
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For  example,  their  first  eommandment,  “One  more  time:  Do  not  believe  in  Silver 
Bullets”  [pg.  3]  applies  to  this  study,  also.  This  has  been  reiterated  in  several  eontexts 
involving  software,  and  eertainly  applies  to  the  use  of  eommereial  item  equipment  in 
programs,  as  well.  Anyone  who  believes  that  seleetion  of  a  eommereial  item  and 
inserting  it  into  a  program  will  be  the  quiek  fix  believes  in  fairy  tales,  and  does  not  really 
understand  the  proeess. 

The  seeond  eommandment,  “Use  the  term  precisely  (and  demand  like  behavior 
from  others)”  [pg.  3]  most  certainly  applies  here.  We  have  noted  in  this  research  that 
terms  have  been  used  differently  in  the  past  in  the  Army  Audit  Agency  reports,  and  that 
Dr.  Gansler  specifically  defined  terms  in  order  to  bring  uniformity  to  the  terminology. 

Their  third  commandment,  “Understand  the  impact  of  COTS  products  on  the 
requirements  and  selection  process”  [pg.  4]  equally  applies  to  this  research.  It  is 
important  that  commercial  products  are  known  and  understood  by  those  who  establish 
requirements.  The  commercial  marketplace  must  be  clearly  understood  in  order  to  have  a 
flexible  range  of  “requirements”  sufficient  to  allow  commercial  items  to  qualify. 

The  fourth  commandment,  “Understand  the  impact  of  COTS  products  on  the 
integration  process,”  [pg.  5]  certainly  applies,  as  demonstrated  by  the  various  integration 
problems  that  have  occurred  within  the  various  services  and  subsequently  reported  in  the 
previously  cited  audits.  There  has  been  a  tendency  to  assume  that  a  commercial  item  can 
be  used  as-is,  without  any  serious  thought  given  to  the  difficulty  and  risk  involved  in  the 
integration  process.  It  has  been  assumed  that  the  use  of  a  commercial  item  alleviates  all 
risk  of  integration.  In  fact,  just  the  opposite  may  occur:  commercial  items  may  be  even 
more  difficult  to  integrate. 

Their  fifth  commandment,  “Understand  their  impact  on  the  testing  process,” 
[pg.  6]  also  applies,  as  has  been  demonstrated  by  the  problems  experienced  in 
commercial  item  testing.  The  validation  of  commercial  items  is  different  from  the  testing 
of  development  projects.  This  difference  in  the  testing  process  must  be  clearly  understood 
in  order  for  validation  of  the  commercial  item  to  be  correctly  planned.  Testing  must  then 
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be  initiated  early  enough  in  the  proeess  in  order  to  alleviate  the  great  risk  that  oeeurs 
when  it  is  performed  too  late. 

Their  sixth  eommandment,  “Realize  that  a  COTS  approaeh  makes  a  system 
dependent  on  the  COTS  vendors,”  [pg.  6]  was  exemplified  in  the  Army  Audit  Ageney 
report  of  11  September  1990.  This  report  found  that  in  most  oases,  for  the  sake  of 
expedienoy,  PMs  opted  for  the  end-item  oontraotor  to  provide  the  maintenanoe  and 
supply  support  for  the  product.  Vendor  support  must  be  examined  critically  for  several 
reasons.  First,  it’s  important  to  insure  that  the  program  doesn’t  become  too  dependent  on 
the  vendor.  Second,  it  must  be  determined  that  the  vendor  has  sufficient  technical  data  for 
adequate  ongoing  support.  Third,  it  must  be  determined  that  the  vendor  has  sufficient 
technical  data  for  reprocurement  of  the  product.  Reprocurement  is  high-risk,  made  so  by 
the  probability  that  the  commercial  item  vendor  may  change  the  product,  or  the  product 
support,  go  out  of  business,  be  bought  out  by  another  company,  etc.  Contingency  plans 
should  therefore  be  developed  for  these  possibilities. 

Their  seventh  commandment,  “Realize  that  maintenance  is  not  free,”  [pg.  7]  is 
exemplified  by  the  Army  Audit  Agency  11  September  1990  report  about  the  blind  faith 
exhibited  by  the  PMO  concerning  the  cost  effective  vendor  support  of  their  products. 
This  audit  report  has  documented  one  of  many  instances  where  no  analysis  was 
performed  to  prove  that  contractor  support  was  more  cost-effective  than  the  use  of  the 
military  in-house  organic  support  system. 

The  eight  commandment,  “You  are  not  absolved  of  the  need  to  engineer  the 

system  well,”  [pg.  8]  has  been  displayed  in  every  one  of  the  issues  studied  in  the  previous 

chapter.  Just  because  a  product  may  be  an  off-the-shelf  commercial  item,  there  is  no 

guarantee  that  it  will  integrate  into  the  program’s  system  successfully.  Nothing  will  come 

together  and  work  by  itself.  The  integration  process  must  not  be  accidental,  but  rather, 

must  be  planned  and  executed  purposefully.  As  has  been  seen  in  the  previous  problems  in 

Chapter  III,  commercial  items  add  another  whole  dimension  to  the  difficulty  of  program 

management.  Elimination  of  systems  engineering  in  the  commercial  item  acquisition 

process  is  not  an  option  with  the  use  of  commercial  items.  As  Carney  notes,  “This  system 

will  need  to  be  designed,  brought  together,  tested,  and  managed  just  the  same  as  any 
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other  system  you  have  built  or  aequired  in  the  past.”  [pg  8]  Logisties  supportability 
planning  and  exeeution  should  be  added  to  Carney’s  list. 

Their  ninth  eommandment,  “Just  ‘doing  COTS’  is  not  an  automatie  eost-saver,” 
[pg  8]  has  been  alluded  to  throughout  this  thesis.  There  has  been  a  lot  of  aneedotal 
evidenee,  as  well  as  a  few  aetual  eases  to  substantiate  the  elaim  that  use  of  eommereial 
items  saves  both  time  and  money.  However,  it  is  not  a  “given”  that  using  a  eommereial 
item  will  always  save  time  or  money.  The  use  of  eommereial  items  must  be  earefully 
planned,  well  engineered,  and  analyzed  in  depth. 

The  tenth  eommandment,  “Just  ‘doing  COTS’  must  be  part  of  a  large-seale 
paradigm  shift,”  [pg.  9]  was  indieated  in  Chapter  II,  paragraph  H,  seetions  1-7,  of  this 
thesis.  Instead  of  the  Traditional  Model,  as  shown  in  Figure  2  of  this  thesis  as  being  a 
waterfall  aequisition  proeess,  it  is  now  imperative  that  proeesses  overlap  and  that  greater 
use  of  IPTs  be  made  in  order  to  manage  the  uniqueness  of  eommereial  item  usage,  as 
portrayed  in  the  Reeommended  Proeess  -  Figure  2. 

D,  COMMERCIAL  ITEM  BEST  PRACTICE  SUMMARY 

The  following  table  summarizes  the  problem/issue  as  a  “risk,”  the  possible  eause 
of  the  unresolved  risk,  and  the  best  praetiee  to  follow  to  minimize  the  risk. 
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Table  3 


Commercial  Item  Best  Practice  Matrix 


RISK  POSSIBLE  CAUSE  BEST  PRACTICE 


Needs  Determination 

-  User  requirements  are 
improperly  identified, 
misunderstood  or  ignored. 

-  Requirements  allowed  to 
grow,  i.e.,  requirements  are 
gold  plated. 

-  Inadequate  acquisition 
planning/risk  analysis 
performed;  inadequate 
training/education;  unclear 
ORD;  IPTs  not  used; 
culture-driven  decisions;  no 
product  standardization;  no 
centralized  database; 
inflexible  reqmts. 

-No  acquisition 
planning/IPT;  needs 
allowed  to  grow  too  easily; 
inadequate  ORD; 
insufficient  market 
research. 

-  Use  IPTs;  clearly  written 

ORD;  comprehensive 
acquisition  planning; 
thorough  market  research; 

adequate  acquisition 
training  and  education;  use 
lessons  learned. 

-  Use  IPTs;  clearly  written 

ORD;  comprehensive 
acquisition  planning; 
thorough  market  research; 
freeze  requirements  or  use 
P^I  process  to  add  needs. 

Acauisition  Stratesv 

-  Incorrect  streamlining 
acquisition  strategy  used. 

-  Lessons  learned  not 
utilized;  no  evaluation  of 
streamlining  decisions; 
inadequate  training/ 
education;  incorrect  culture- 
driven  decisions. 

-  Utilize  lessons  learned, 
closely  evaluate  stream¬ 
lining  decisions;  provide 
adequate  acquisition 
training  and  education;  need 
comprehensive  review  of  & 
change  of  culture  paradigm. 

Market  Investigations 

-  Inadequate  or  no  market 
investigation  performed. 

-  No  early,  robust,  market 
investigation  performed; 
inadequate 

training/education;  no 
centralized  database/lessons 
learned. 

-  Perform  robust  market 
investigation  early  in  the 
acquisition  process; 
adequately  trained 
personnel;  use  centralized 
database  +  lessons  learned. 

Concept  Formulation 

-  Inadequate  or  no  concept 
formulation  package 
prepared. 

-  Inadequate  training/ 
education/  or  lessons 
learned;  incorrect  culture- 
driven  decisions. 

-Use  lessons  learned; 
adequately  trained  and 
educated  personnel;  need 
comprehensive  review  of  & 
change  of  culture  paradigm. 
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Table  3  (continued) 


Commercial  Item  Best  Practice  Matrix 


RISK  POSSIBLE  CAUSE  BEST  PRACTICE 


Risk  Analysis 

-  A  risk  analysis  was  not 
performed. 

-  No  early,  robust,  market 
investigation  performed; 
inadequate  training  / 
education  /  or  use  of  lessons 
learned;  widespread 
misunderstanding  of  the 
complexity  of  integration. 

-  Perform  robust  market 
investigation  early  in  the 
systems  acquisition  process; 
adequate  acquisition 
training  and  education; 
centralized  database  /  use 
lessons  learned. 

Manage  Integration 

-  Inaeeurate  assessment  of 
program  integration  risk. 

-  No  early,  robust,  market 
investigation  performed; 
inadequate  training  / 
education  /  or  use  of  lessons 
learned;  widespread 
misunderstanding  of  the 
complexity  of  integration. 

-  Perform  robust  market 
investigation  early  in  the 
systems  acquisition  process; 
adequately  trained  and 
educated  personnel; 
centralized  database  /  use 
lessons  learned. 

Test  &  Evaluation 

-  Testing  was  inadequate  or 
was  not  performed. 

-  Culture-driven  poor 
decisions;  inadequate 
training  /  education  /  or  use 
of  lessons  learned;  lack  of 
funding;  lack  of  resources. 

-  change  the  culture’s 
paradigm;  adequately  train 
and  educate  personnel; 
create  centralized  database  / 
use  lessons  learned. 

Design 

-  The  design  was 
underestimated. 

-  The  design  proeess  was 
ineorrectly  aceelerated. 

-  Inadequate  training  / 
education  /  or  use  of  lessons 
learned;  culture-driven  poor 
decision. 

-  Culture-driven;  inadequate 
training/education/use  of 
lessons  learned. 

-  change  the  culture’s 
paradigm;  adequately  train 
and  educate  personnel; 
create  centralized  database  / 
use  lessons  learned. 

Reliability 

-  The  program  reliability 

was  inadequate. 

-  No  quantitative  reliability 
requirements  established. 

-  Contractor’s  reliability 
estimates  not  verified. 

-  Inadequate  reliability 
testing  prior  to  contract 

award. 

-  Poor  culture-driven 
decisions;  inadequate 
training  /  education  /  use  of 
lessons  learned,  lack  of 
funding. 

-  Change  the  culture’s 
paradigm;  adequately  train 
and  educate  personnel; 
create  centralized  database  / 
use  lessons  learned. 
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Table  3  (continued) 


Commercial  Item  Best  Practice  Matrix 


RISK 

POSSIBLE  CAUSE 

BEST  PRACTICE 

Support  Requirements 

-  Incorrect  program  support 
concept  is  selected. 

-  Poor  culture-driven 
decisions;  inadequate 
training  /  education  /  use  of 
lessons  learned; 
misunderstanding  of  the 
schedule  needed  for 
logistics  even  for  CI/NDI 
items. 

-  Change  culture’s 
paradigm;  adequately  train 
and  educate  personnel; 
create  centralized  database  / 
use  lessons  learned. 

-  Inadequate  logistics 
support  is  planned. 

-  Insufficient  time  allotted 
for  logistics  support 
planning  /  preparation; 
EMD  phase  eliminated. 

-  Stretch  phase  to 
accommodate  logistics 
support  planning  such  as 
TMs,  training,  provisioning. 

Tech  Data  -  Competitive 
Reprocurement 

-  Data  adequate  for 
reprocurement  is  not 
obtained. 

-  Poor  culture-driven 
decisions;  inadequate 
training  /  education  /  use 
of  lessons  learned. 

-  Change  culture’s 
paradigm;  adequately  train 
and  educate  personnel; 
create  centralized  database  / 
use  lessons  learned 

E.  CONCLUSION 

In  order  to  reduce  the  time  and  cost  of  fielding  new  systems  and  inserting  new 
technologies  rapidly,  the  well  thought  out  use  of  commercial  items  is  imperative. 
However,  the  difficult  new  paradigm  of  using  commercial  items  must  be  recognized,  and 
acquisition  personnel  educated  and  trained  for  its  efficient  and  effective  use.  The 
previous  table  of  best  practices  is  provided  in  order  to  highlight  the  major  areas  of  risk 
that  have  been  found  to  exist  within  the  acquisition  community.  It  also  summarizes  the 
best  practices  that  will  serve  to  minimize  those  risks. 
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F.  AREAS  FOR  FURTHER  RESEARCH 

This  thesis  identified  areas  that  merit  additional  researeh,  but  did  not  address  them 
beeause  they  were  beyond  the  seope  of  this  study.  Those  areas  are  as  follows: 

1 .  An  in-depth  analysis  of  the  eost-benefit  relationship  of  using  eommereial 
items  in  aequisition  programs. 

2.  An  examination  of  any  existing  barriers  to  using  eommereial  items  more 
efficiently  in  DoD  acquisition  programs. 

3.  A  comparative  analysis  of  the  problems  associated  with  using  commercial 
items  versus  the  problems  found  in  traditional  acquisition  programs  to  determine  the 
significant  differences,  if  any. 

4.  An  analysis  of  major  defense  acquisition  programs’  potential  of  meeting 
the  OSD  commercial  item  goals  that  have  been  set  for  DoD. 

5.  Identification  of  the  extent  the  current  PM  community  acquisition  culture 
affects  the  PM’s  correct  usage  of  commercial  items,  the  extent  to  which  it  contributes  to 
the  problems,  and  recommended  corrective  actions. 

6.  Identification,  via  survey  or  other  means,  of  the  extent  to  which  the 
“culture”  in  which  the  PM  is  immersed  drives  incorrect  or  inadequate  decisions  within 
the  acquisition  process  and  potential  corrective  actions. 


73 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


74 


LIST  OF  REFERENCES 


Air  Force  Space  and  Missile  Command.  CPATs  -  Reliability  Engineering,  14  August 
1998.  Military  Specifieations  and  Standards  Reform  Program  (MSSRP),  Critical  Process 
Assessment  Tool  (CP AT),  SMC/AXM.  Retrieved  16  Oetober  2001  from  the  World 
Wide  Web;  http://web2.deskbook.osd.miFdata/012GZDOC.DOC. 

Anderson,  Robert  H.  and  Hundley,  Riehard  O.  The  Implications  of  COTS  Vulnerabilities 
for  the  DoD  and  Critical  U.S.  Infrastructures:  What  Can/Should  the  DoD  Do?.  RAND. 
1998. 

ASAAL&T.  Aeting  Assistant  Seeretary  of  the  Army  for  Aequisition,  Logistics  and 
Technology.  Memorandum  Subject  Commercial  Acquisitions.  26  March  2001. 

Baker,  Don.  Commercial  Off-The-Shelf  Armed  Forees  Journal  International.  November, 
1997.  Retrieved  21  July  2001  from  the  World  Wide  Web; 
http;//www.afii.eom/mags/1997/Nov/Featurel.html. 

Bonheim,  Mike  FTC.  PM  -  Physical  Security  Equipment.  Commercial  Items  Acquisition 
Data  Call  S:  COB  22  February  2001. 

Brinkley,  C.  Mark.  We  Need  to  Lie.  Marine  Corps  Times,  pgs  8-10.  29  January  2001. 

Carney,  D.  and  Obemdorf,  P.  OSD  -  Commandments  of  COTS:  Still  in  Search  of  the 
Promised  Land.  Software  Engineering  Institute,  Carnegie  Mellon  University.  Retrieved  4 
September  2001  from  the  World  Wide  Web; 
http;//www.deskbook.osd.mil/irp/253.asp  -  MandRef 

CJCS.  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  5261. 01 A.  I  August  1998. 
Retrieved  24  September  2001  from  the  World  Wide  Web; 
http;//www. datashopper.dk/~boo/eomterfund.html. 

Defense  Systems  Management  College.  Systems  Engineering  Management  Guide. 
DSMC,  Ft.  Belvoir,  VA.  January  1990.  pg  13-1 

DoD.  DoDD  5000.1.  The  Defense  Acquisition  System,  paragraph  4.2.3,  (Incorporating 
Change  I,  4  January  2001)  23  October  2000.  Retrieved  20  August  2001  from  the  World 
Wide  Web;  http://web2.deskbook.osd.mi1/remb/MDOD/Q61DD/061DDDOC.HTM#T2. 

DoD.  DoDI  5000.2.  Operation  of  the  Defense  Aequisition  System;  (Including  Change  1); 
4  January  2001.  Retrieved  23  August  2001  from  the  World  Wide  Web; 
http://webl.deskbook.osd.miFhtmlfiles/rlframe/REFLIB  Frame.asp?TOCAhtmlflles/TOC/012ditoe.htm& 

Doe=/reflib/mdod/Q  1 2di/Q  1 2didoe.htm&BMK-T  19. 


75 


DoD  IG.  Department  of  Defense  Inspector  General.  Allegations  Involving  the 
Procurement  of  the  Hunter  Unmanned  Aerial  Vehicle:  Short-Range  System.  Report  No. 
96-111.  7  May  1996.  Retrieved  26  September  2001  from  the  World  Wide  Web; 
http://www.dodig.osd.mil/audit/reports/96report.htm. 

DoD  IG.  Department  of  Defense  Inspector  General.  DoD  Hotline  Allegation  Regarding 
the  Naval  Special  Warfare  Patrol  Coastal  Ship  and  Rigid  Inflatable  Boat  Acquisition 
Programs.  Report  95-193.  15  May  1995. 

DoD  IG.  Department  of  Defense  Inspector  General.  Effectiveness  of  DoD  Use  of 
Nondevelopmental  Items  in  Major  Defense  Acquisition  Programs.  Report  No.  92-107.  22 
June  1992.  Retrieved  26  September  2001  from  the  World  Wide  Web; 
http://www.dodig.osd.mil/audit/reports/92report.htm. 

DoD  IG.  Department  of  Defense  Inspector  General.  Lessons  Learned  from  Acquisitions 
of  Modified  Commercial  Items  and  Nondevelopmental  Items.  Report  No.  97-219.  23 
September  1997. 

DoD.  Deputy  Under  Secretary  of  Defense  (Acquisition  Reform).  Draft  Commercial  Item 
Handbook.  10  January  2001.  Retrieved  15  August  2001  from  the  World  Wide  Web; 
http://www.aca.osd.mil/cgi-bin/web  evaluate?dataset= 

ar&dbs=ar&multi=l&auerv=Commercial+Item+  Policv&searchButton=Search. 

DoD.  Office  of  the  Deputy  Under  Secretary  of  Defense  for  Acquisition  Reform. 
Commercial  Item  Acquisition:  Considerations  and  Lessons  Learned.  14  July  2000. 

DoD.  Under  Secretary  of  Defense  for  Acquisition  and  Technology.  Buying  Commercial 
&  NonDevelopmental  Items:  A  Handbook.  April  1996. 

DoD.  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics.  Dr.  J.  S. 
Gansler.  Policy  Memorandum:  Commercial  Acquisition.  5  January  2001. 

Federal  Acquisition  Regulation  Parts  2  and  12,  (25  June  2001).  Retrieved  15  August 
2001  from  the  World  Wide  Web:  http://www.arnet.gov/far/loadmain.html. 

Federal  Acquisition  Reform  Act.  Title  10.  Section  2304  of  Public  Law  104-121.  1996. 

Federal  Acquisition  Streamlining  Act.  Title  10.  Section  8104,  paragraph  2377  of  Public 
Law  103-355.  1994. 

Flack,  Marla.  AMSEL-LG-CM.  Impact  of  DA  Policy  on  Commercial  Items.  Information 
Paper.  18  May  2001. 

General  Accounting  Office.  GAO/NSIAD-00-199.  Best  Practices:  A  More  Constructive 
Test  Approach  Is  Key.  31  July  2000.  Retrieved  30  July  2001  from  the  World  Wide  Web; 

http://www.gao.gov/. 


76 


General  Aeeounting  Offiee.  GAO/NSIAD-95-161.  UNMANNED  AERIAL  VEHICLES: 
Maneuver  System  Schedule  Includes  Unnecessary  Risk.  September  1995.  Retrieved  25 
September  2001  from  the  World  Wide  Web:  http://www.gao.gov/. 


General  Accounting  Office.  GAO/NSIAD-96-2,  Unmanned  Aerial  Vehicles:  Hunter 
System  Is  Not  Appropriate  for  Navy  Fleet  Use.  1  December  1995.  Retrieved  25 
September  2001  from  the  World  Wide  Web:  http://www.gao.gov/. 

Kang,  Keebom.  DoD  Inventory  Management  Culture  Changes  and  Training  in 
Commercial  Practices.  Research  paper  NPS-SM-98-002.  March  1998. 

U.S.  Army  Audit  Agency.  Acquisition  of  Nondevelopmental  Items  US.  Army  Natick 
Research,  Development  and  Engineering  Center  Natick,  Massachusetts.  (Report  of 
Audit  NE  90-206).  19  March  1990. 

U.S.  Army  Audit  Agency.  Acquisition  of  Nondevelopmental  Items.  (Report  of  Audit  NE 
91-204).  17  June  1991. 

U.S.  Army  Audit  Agency.  Acquisition  of  Nondevelopmental  Items,  Fort  Monmouth,  New 
Jersey.  (Report  of  Audit  NE  90-209).  1 1  September  1990. 


77 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


78 


BIBLIOGRAPHY 


DoD  IG.  Department  of  Defense  Inspeetor  General.  Contractor ’s  Performance  on  the 
Family  of  Medium  Tactical  Vehicles  Program.  Report  96-020.  November  1995. 

DoD  IG.  Department  of  Defense  Inspeetor  General.  Purchasing  Commercial  Products. 
Report  97-145.  May  1997. 

DoDR  5000.2.  Department  of  Defense  Regulation.  Mandatory  Procedures  for  Major 
Defense  Acquisition  Programs  (MDAPS)  and  Major  Automated  Information  System 
(MAIS)  Acquisition  Programs.  DoD  Deskbook,  10  June  2001. 

DoN.  Department  of  the  Navy.  NAVSEA  —  Acquisition  Planning  Guide.  Deeember  1999. 

GAO.  General  Accounting  Office.  ACQUISITION  REFORM  -  The  Government’s  Market 
Research  Efforts.  GAO/NSI AD-97-3.  October  1996. 

GAO.  General  Accounting  Office.  ACQUISITION  REFORM  -  Military-Commercial 
Pilot  Program  Offers  Benefits  but  Faces  Challenges,  Government  Accounting  Office. 
GAO/NSIAD-96-53.  June  1996. 

GAO.  General  Accounting  Office.  ARMY  ACQUISITION  -  Medium  Trucks  Passed  Key 
Operational  and  Technical  Tests,  Government  Accounting  Office.  GAO/NSI  AD-96-4. 
January  1996. 

Oberndorf,  Tricia,  Brownsword,  Lisa  and  Sledge,  Carol  A.  An  Activity  Framework  for 
COTS-Based  Systems.  Technical  Report  CMU/SEI-2000-TR-010.  Carnegie  Mellon 
University.  October  2000. 

Oberndorf,  Tricia.  COTS-Based  Systems  (CBS)  Initiative.  Carnegie  Mellon  University. 
Retrieved  23  August  2001  from  the  World  Wide  Web; 
http://www.sei.cmu.edu/cbs/cbs  description.html. 

USAAA.  U.S.  Army  Audit  Agency.  Acquisition  of  Nondevelopmental  Items  —  Program 
Executive  Officer,  Combat  Support  and  U.S.  Army  Tank-Automotive  Command,  Warren, 
Michigan,  Report  No.  EC-90-211,  21  August  1990. 


79 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


80 


LIST  OF  ACRONYMS  AND  ABBREVIATIONS 


ASA(AT&L) 

Assistant  Secretary  of  the  Army  for  Acquisition  Technology  and 

Logistics 

Cl 

Commercial  Item 

CINC 

Commander  in  Chief  of  a  Unified  Command 

CNO 

Chief  of  Naval  Operations 

COTS 

Commercial  Off-The-Shelf 

DoD 

Department  of  Defense 

DoDD 

Department  of  Defense  Directive 

DoDI 

Department  of  Defense  Instruction 

DoDR 

Department  of  Defense  Regulation 

DT 

Developmental  Test 

DT&E 

Developmental  Test  &  Evaluation 

DUSD(AR) 

Deputy  Under  Secretary  of  Defense  for  Acquisition  Reform 

EMD 

Engineering  &  Manufacturing  Development 

EAR 

Eederal  Acquisition  Regulation 

EASA 

Eederal  Acquisition  Streamlining  Act 

GOTS 

Government  Off-the-Shelf 

GSA 

General  Services  Administration 

GSS 

General  Ships  Specifications 

HQAMC 

Headquarters  of  the  Army  Material  Command 

lEB 

Invitation  for  Bid 

IPPD 

Integrated  Product  and  Process  Development 

IPT 

Integrated  Process  Team 

ECC 

Life  Cycle  Cost 

ERIP 

Low-Rate  Initial  Production 

MIL-SPEC 

Military  Specification 

MOA/MOU 

Memorandum  of  Agreement/Understanding 

NDI 

Nondevelopmental  Item 

NSSC 

Naval  Sea  Systems  Command 
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NSWC 

Naval  Surface  Warfare  Center 

ORD 

Operational  Requirements  Doeument 

OT 

Operational  Test 

OT&E 

Operational  Test  &  Evaluation 

PEO 

Program  Exeeutive  Offieer 

PM 

Program  Manager 

p3i 

Preplanned  Produet  Improvement 

R&D 

Researeh  and  Development 

RDT&E 

Researeti,  Development,  Test  &  Evaluation 

REP 

Request  for  Proposal 

REQ 

Request  for  Quotation 

RIB 

Rigid  Inflatable  Boat 

R&M 

Reliability  and  Maintainability 

SDD 

System  Development  and  Demonstration 

T&E 

Test  and  Evaluation 

USAAA 

U.S.  Army  Audit  Ageney 

use 

United  States  Code 

USD(AT&E) 

Under  Seeretary  of  Defense  for  Aequisition,  Teehnology  and 

Eogisties 

UE 

Underwriters  Eaboratory 
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